BioCASe Provider Software wikidb https://wiki.bgbm.org/bps/index.php/Main_Page MediaWiki 1.28.2 first-letter Media Special Talk User User talk BioCASe Provider Software BioCASe Provider Software talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk 0 1 1 2011-01-06T13:52:01Z MediaWiki default 0 wikitext text/x-wiki '''MediaWiki has been successfully installed.''' Consult the [http://meta.wikimedia.org/wiki/Help:Contents User's Guide] for information on using the wiki software. == Getting started == * [http://www.mediawiki.org/wiki/Manual:Configuration_settings Configuration settings list] * [http://www.mediawiki.org/wiki/Manual:FAQ MediaWiki FAQ] * [https://lists.wikimedia.org/mailman/listinfo/mediawiki-announce MediaWiki release mailing list] 25b5c21bea6319ae81e1379d4e59bdfe9100ce36 6 1 2011-06-16T11:21:53Z JoergHoletschek 3 wikitext text/x-wiki = BioCASe Provider Software = bla = Table of Conetens = blubb 8ed7baf7f60586abd0c5c3fe912ed869f0c6629b 7 6 2011-06-16T11:23:01Z JoergHoletschek 3 wikitext text/x-wiki asdfasdfasfd = BioCASe Provider Software = bla = Table of Conetens = blubb 1c2ea101060a9c5769c06599fe5b135f8522bc85 8 7 2011-06-16T11:24:04Z JoergHoletschek 3 wikitext text/x-wiki asdfasdfasfd asdf '''asdf''' asdf ''asdf'' = BioCASe Provider Software = bla = Table of Conetens = == Table of Conetens == == Table of Conetens == blubb 8fc5c1d9722e0ba16c1ca1d5f677869ffd6b64a6 9 8 2011-06-16T11:41:26Z JoergHoletschek 3 wikitext text/x-wiki asdfasdfasfd asdf '''asdf''' asdf ''asdf'' = BioCASe Provider Software Wiki = bla == Table of Conetens == == asdfsds == blubb d2a00000be2bbecef005a439fb24d4868925ecee 10 9 2011-06-16T11:56:53Z JoergHoletschek 3 Reverted edits by [[Special:Contributions/JoergHoletschek|JoergHoletschek]] ([[User talk:JoergHoletschek|talk]]) to last revision by [[User:MediaWiki default|MediaWiki default]] wikitext text/x-wiki '''MediaWiki has been successfully installed.''' Consult the [http://meta.wikimedia.org/wiki/Help:Contents User's Guide] for information on using the wiki software. == Getting started == * [http://www.mediawiki.org/wiki/Manual:Configuration_settings Configuration settings list] * [http://www.mediawiki.org/wiki/Manual:FAQ MediaWiki FAQ] * [https://lists.wikimedia.org/mailman/listinfo/mediawiki-announce MediaWiki release mailing list] 25b5c21bea6319ae81e1379d4e59bdfe9100ce36 8 2 2 2011-03-16T12:24:47Z WikiSysop 1 Created page with "/* CSS placed here will be applied to all skins */ /* Hide title of Main_Page */ /* body.page-Main_Page h1.firstHeading { display:none; } */ /* Hide every title except special ..." css text/css /* CSS placed here will be applied to all skins */ /* Hide title of Main_Page */ /* body.page-Main_Page h1.firstHeading { display:none; } */ /* Hide every title except special pages */ body.ns-subject h1.firstHeading { display:none; } 8ee4467e7126608246798d0e6de2da479f439d5e 10 5 5 2011-03-17T13:23:22Z WikiSysop 1 wikitext text/x-wiki <div style="padding:0.3em 0.5em 0.3em 0.5em; border-style:solid; border-width:medium; border-color:#ffdd00"> <span style="color:#cf7606; font-size:140%;">'''Edit this page'''</span> __NOTOC__ This "sandbox" page is to allow you to carry out experiments. Please feel free to try your skills at formatting here. If you want to learn more about how to edit a wiki, please read [http://www.mediawiki.org/wiki/Help:Editing_pages this introduction] or the [[Wikipedia:Wikipedia:Tutorial|tutorial at Wikipedia]]. __NOEDITSECTION__ To edit, click ''"edit"'' at the top of the page, make your changes in the dialog box, and click the ''"Save page"'' button when you are finished. Please refrain from adding material that is any way offensive, that is copyrighted, or that is at all libelous. Please do '''NOT''' create new pages without any meaningful content just to try it out! </div> = Testing Area = 3ef9536f38b2357328bd1ea5c92e37389a2cab2c 8 3 3 2011-03-17T13:56:56Z WikiSysop 1 Created page with "* navigation ** mainpage|mainpage-description ** portal-url|portal ** currentevents-url|currentevents ** recentchanges-url|recentchanges ** randompage-url|randompage ** mediawiki..." wikitext text/x-wiki * navigation ** mainpage|mainpage-description ** portal-url|portal ** currentevents-url|currentevents ** recentchanges-url|recentchanges ** randompage-url|randompage ** mediawikiwiki:Help:Contents|help * SEARCH * TOOLBOX * LANGUAGES 3815dfdeba7a9affe9eeda9aa612a0c227e89ce5 0 4 4 2011-03-18T10:55:16Z WikiSysop 1 wikitext text/x-wiki {{Sandbox}}<!--Please leave this line alone and write below (this is the coloured heading)--> == Formatting == ''italic'' '''bold''' == Linking == http://purl.net/wiki/moin/ [http://www.python.org/ Python] [http://c2.com/sig/wiki.gif] == Lists == # Numbered lists are good ## very organized ## easy to follow === Bullet === * Lists are easy to do: ** start every line * with a star ** more stars mean *** deeper levels Here is a '''new''' test 40aa51520aadcc9711ee71b7d769660859a44f3b 0 6 11 2011-06-21T11:06:49Z JoergHoletschek 3 Created page with "This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please re..." wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the ___Introduction___ and the ___Beginner’s Guide___ first. Requirements Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: 1. You have administrative privileges on this machine, i.e. the user rights to install new software packages. 2. It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. 3. A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e.g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the ___FAQs___ or ___get in contact with the BioCASE Team___. Installing Apache/IIS and Python If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the ___libs test page___ of the BGBM installation, find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. Downloading the Provider Software You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. Downloading the Archive File Just download the archive from the ___download section of the BioCASe site___ and unpack the contents to the destination folder. Downloading from the Subversion Repository If you’re using Windows, you need to install a Subversion client first (try __Tortoise___, which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase. This will connect to our server with anonymous access and download all files to the folder biocase. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose Checkout. Fill in the URL, change the destination folder name into something meaningful and press OK: For the remainder of this guide, we will assume you’ve installed the BPS into a folder named biocase, which should look like this by now: Running the Setup Script The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (‘’’biocase’’’) and type in python setup.py (in case Python is not found, you’ll have to add the full path name to the Python executable). Make sure the Python version given at the beginning out the output is the one you desire (otherwise add the path to the correct executable): All of the questions do have defaults. If you want to keep them, just press Enter. Import configuration from existing installation. Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. Webserver domain: This is the domain name of the web server you’re installing BioCASe on, e.g. http//www.foobar.org. You can keep the default localhost as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. Base URL: This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at www.foobar.org/biocase, for example. After answering these three questions, the script will update your files. At the end of the output it will print the lines you will need later to create the link, so do not close the window! Linking BioCASe and Your Web Server The public part of the provider software is stored in the www subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so will compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in C:\biocase, the folder that must be published with a Virtual directory is C:\biocase\www and will be accessible later through http://www.foobar.org/biocase. If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file httpd.conf. This can typically be found in the conf folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! Save the file and restart Apache for the changes to take effect. If you use Internet Information Server, you need to create a virtual directory for the folder c:\biocase\www using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the “Configure” button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add index.html to the list of default documents (virtual directory > properties > documents > default). We know that configuring Python for IIS can be cumbersome; however, there are several good tutorials on the web, for example http://www.ecp.cc/pyiis.html http://support.microsoft.com/default.aspx?scid=kb%3Ben-us%3B276494 http://python.markrowsoft.com/iiswse.asp. Once you’re done, you can test your BioCASe Installation by opening a browser and typing in http://localhost/biocase (replace localhost with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: Installing Additional Python Packages Before you can start configuring your installation, you need to install additional packages, depending on the DBMS used by the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on ‘Utilities’, then “library test” in order to navigate to he library test page that shows the status of installed libraries. On a vanilla BPS installation, it should look similar to this: The BioCASe libraries ship with the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. Changing the Default Password The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking “Start” at the top of the page, or by retyping http://localhost/biocase in the browser window), choose “Config Tool”, then “System Administration”. You will then be prompted for a password. Type in the default ACDC, then click “Submit”. The top of the configuration page you’ll be directed to should look similar to this: In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults localhost and biocase). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press “update config” to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled “Graphviz dot binary”. On Linux machines it might look similar to the default (/usr/local/bin/dot), on Windows machines it will be probably something like C:\Program Files\Graphviz2.26.3\bin\dot.exe. Press “update config” to save the path. Checking Your Installation Before you proceed to the configuration, you should make sure everything is installed correctly: 1. Typing in http://localhost/biocase (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown here) 2. The libraries test page (Utilities/library test) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). 3. “Status of writeable directories” shows a green writeable for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: 1e4d40b77fb4f4eabcfcf082dceca00d25948dba 12 11 2011-06-21T11:11:59Z JoergHoletschek 3 wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the ___Introduction___ and the ___Beginner’s Guide___ first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e.g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the ___FAQs___ or ___get in contact with the BioCASE Team___. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the ___libs test page___ of the BGBM installation, find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. Downloading the Archive File Just download the archive from the ___download section of the BioCASe site___ and unpack the contents to the destination folder. Downloading from the Subversion Repository If you’re using Windows, you need to install a Subversion client first (try __Tortoise___, which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase. This will connect to our server with anonymous access and download all files to the folder biocase. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose Checkout. Fill in the URL, change the destination folder name into something meaningful and press OK: For the remainder of this guide, we will assume you’ve installed the BPS into a folder named biocase, which should look like this by now: ==Running the Setup Script== The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (‘’’biocase’’’) and type in python setup.py (in case Python is not found, you’ll have to add the full path name to the Python executable). Make sure the Python version given at the beginning out the output is the one you desire (otherwise add the path to the correct executable): All of the questions do have defaults. If you want to keep them, just press Enter. Import configuration from existing installation. Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. Webserver domain: This is the domain name of the web server you’re installing BioCASe on, e.g. http//www.foobar.org. You can keep the default localhost as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. Base URL: This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at www.foobar.org/biocase, for example. After answering these three questions, the script will update your files. At the end of the output it will print the lines you will need later to create the link, so do not close the window! ==Linking BioCASe and Your Web Server== The public part of the provider software is stored in the www subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so will compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in C:\biocase, the folder that must be published with a Virtual directory is C:\biocase\www and will be accessible later through http://www.foobar.org/biocase. If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file httpd.conf. This can typically be found in the conf folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! Save the file and restart Apache for the changes to take effect. If you use Internet Information Server, you need to create a virtual directory for the folder c:\biocase\www using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the “Configure” button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add index.html to the list of default documents (virtual directory > properties > documents > default). We know that configuring Python for IIS can be cumbersome; however, there are several good tutorials on the web, for example http://www.ecp.cc/pyiis.html http://support.microsoft.com/default.aspx?scid=kb%3Ben-us%3B276494 http://python.markrowsoft.com/iiswse.asp. Once you’re done, you can test your BioCASe Installation by opening a browser and typing in http://localhost/biocase (replace localhost with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: ==Installing Additional Python Packages== Before you can start configuring your installation, you need to install additional packages, depending on the DBMS used by the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on ‘Utilities’, then “library test” in order to navigate to he library test page that shows the status of installed libraries. On a vanilla BPS installation, it should look similar to this: The BioCASe libraries ship with the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. ==Changing the Default Password== The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking “Start” at the top of the page, or by retyping http://localhost/biocase in the browser window), choose “Config Tool”, then “System Administration”. You will then be prompted for a password. Type in the default ACDC, then click “Submit”. The top of the configuration page you’ll be directed to should look similar to this: In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults localhost and biocase). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press “update config” to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled “Graphviz dot binary”. On Linux machines it might look similar to the default (/usr/local/bin/dot), on Windows machines it will be probably something like C:\Program Files\Graphviz2.26.3\bin\dot.exe. Press “update config” to save the path. ==Checking Your Installation== Before you proceed to the configuration, you should make sure everything is installed correctly: 1. Typing in http://localhost/biocase (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown here) 2. The libraries test page (Utilities/library test) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). 3. “Status of writeable directories” shows a green writeable for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: e98ca1335e4ec18773495afb905bb0e34657027b 13 12 2011-06-21T11:22:50Z JoergHoletschek 3 /* Installing Apache/IIS and Python */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the ___Introduction___ and the ___Beginner’s Guide___ first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e.g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the ___FAQs___ or ___get in contact with the BioCASE Team___. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. Downloading the Archive File Just download the archive from the ___download section of the BioCASe site___ and unpack the contents to the destination folder. Downloading from the Subversion Repository If you’re using Windows, you need to install a Subversion client first (try __Tortoise___, which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase. This will connect to our server with anonymous access and download all files to the folder biocase. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose Checkout. Fill in the URL, change the destination folder name into something meaningful and press OK: For the remainder of this guide, we will assume you’ve installed the BPS into a folder named biocase, which should look like this by now: ==Running the Setup Script== The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (‘’’biocase’’’) and type in python setup.py (in case Python is not found, you’ll have to add the full path name to the Python executable). Make sure the Python version given at the beginning out the output is the one you desire (otherwise add the path to the correct executable): All of the questions do have defaults. If you want to keep them, just press Enter. Import configuration from existing installation. Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. Webserver domain: This is the domain name of the web server you’re installing BioCASe on, e.g. http//www.foobar.org. You can keep the default localhost as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. Base URL: This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at www.foobar.org/biocase, for example. After answering these three questions, the script will update your files. At the end of the output it will print the lines you will need later to create the link, so do not close the window! ==Linking BioCASe and Your Web Server== The public part of the provider software is stored in the www subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so will compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in C:\biocase, the folder that must be published with a Virtual directory is C:\biocase\www and will be accessible later through http://www.foobar.org/biocase. If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file httpd.conf. This can typically be found in the conf folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! Save the file and restart Apache for the changes to take effect. If you use Internet Information Server, you need to create a virtual directory for the folder c:\biocase\www using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the “Configure” button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add index.html to the list of default documents (virtual directory > properties > documents > default). We know that configuring Python for IIS can be cumbersome; however, there are several good tutorials on the web, for example http://www.ecp.cc/pyiis.html http://support.microsoft.com/default.aspx?scid=kb%3Ben-us%3B276494 http://python.markrowsoft.com/iiswse.asp. Once you’re done, you can test your BioCASe Installation by opening a browser and typing in http://localhost/biocase (replace localhost with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: ==Installing Additional Python Packages== Before you can start configuring your installation, you need to install additional packages, depending on the DBMS used by the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on ‘Utilities’, then “library test” in order to navigate to he library test page that shows the status of installed libraries. On a vanilla BPS installation, it should look similar to this: The BioCASe libraries ship with the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. ==Changing the Default Password== The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking “Start” at the top of the page, or by retyping http://localhost/biocase in the browser window), choose “Config Tool”, then “System Administration”. You will then be prompted for a password. Type in the default ACDC, then click “Submit”. The top of the configuration page you’ll be directed to should look similar to this: In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults localhost and biocase). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press “update config” to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled “Graphviz dot binary”. On Linux machines it might look similar to the default (/usr/local/bin/dot), on Windows machines it will be probably something like C:\Program Files\Graphviz2.26.3\bin\dot.exe. Press “update config” to save the path. ==Checking Your Installation== Before you proceed to the configuration, you should make sure everything is installed correctly: 1. Typing in http://localhost/biocase (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown here) 2. The libraries test page (Utilities/library test) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). 3. “Status of writeable directories” shows a green writeable for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: d9fe6c61d81a7866b99e2a3c3e720dd1a3f937a9 14 13 2011-06-21T11:35:58Z JoergHoletschek 3 /* Downloading the Provider Software */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the ___Introduction___ and the ___Beginner’s Guide___ first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e.g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the ___FAQs___ or ___get in contact with the BioCASE Team___. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase. This will connect to our server with anonymous access and download all files to the folder biocase. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose Checkout. Fill in the URL, change the destination folder name into something meaningful and press OK: For the remainder of this guide, we will assume you’ve installed the BPS into a folder named biocase, which should look like this by now: ==Running the Setup Script== The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (‘’’biocase’’’) and type in python setup.py (in case Python is not found, you’ll have to add the full path name to the Python executable). Make sure the Python version given at the beginning out the output is the one you desire (otherwise add the path to the correct executable): All of the questions do have defaults. If you want to keep them, just press Enter. Import configuration from existing installation. Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. Webserver domain: This is the domain name of the web server you’re installing BioCASe on, e.g. http//www.foobar.org. You can keep the default localhost as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. Base URL: This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at www.foobar.org/biocase, for example. After answering these three questions, the script will update your files. At the end of the output it will print the lines you will need later to create the link, so do not close the window! ==Linking BioCASe and Your Web Server== The public part of the provider software is stored in the www subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so will compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in C:\biocase, the folder that must be published with a Virtual directory is C:\biocase\www and will be accessible later through http://www.foobar.org/biocase. If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file httpd.conf. This can typically be found in the conf folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! Save the file and restart Apache for the changes to take effect. If you use Internet Information Server, you need to create a virtual directory for the folder c:\biocase\www using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the “Configure” button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add index.html to the list of default documents (virtual directory > properties > documents > default). We know that configuring Python for IIS can be cumbersome; however, there are several good tutorials on the web, for example http://www.ecp.cc/pyiis.html http://support.microsoft.com/default.aspx?scid=kb%3Ben-us%3B276494 http://python.markrowsoft.com/iiswse.asp. Once you’re done, you can test your BioCASe Installation by opening a browser and typing in http://localhost/biocase (replace localhost with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: ==Installing Additional Python Packages== Before you can start configuring your installation, you need to install additional packages, depending on the DBMS used by the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on ‘Utilities’, then “library test” in order to navigate to he library test page that shows the status of installed libraries. On a vanilla BPS installation, it should look similar to this: The BioCASe libraries ship with the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. ==Changing the Default Password== The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking “Start” at the top of the page, or by retyping http://localhost/biocase in the browser window), choose “Config Tool”, then “System Administration”. You will then be prompted for a password. Type in the default ACDC, then click “Submit”. The top of the configuration page you’ll be directed to should look similar to this: In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults localhost and biocase). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press “update config” to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled “Graphviz dot binary”. On Linux machines it might look similar to the default (/usr/local/bin/dot), on Windows machines it will be probably something like C:\Program Files\Graphviz2.26.3\bin\dot.exe. Press “update config” to save the path. ==Checking Your Installation== Before you proceed to the configuration, you should make sure everything is installed correctly: 1. Typing in http://localhost/biocase (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown here) 2. The libraries test page (Utilities/library test) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). 3. “Status of writeable directories” shows a green writeable for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: b27f3f7d426d5964b369a123337582e801ca7105 15 14 2011-06-21T14:56:56Z JoergHoletschek 3 /* Downloading the Provider Software */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the ___Introduction___ and the ___Beginner’s Guide___ first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e.g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the ___FAQs___ or ___get in contact with the BioCASE Team___. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder <small>biocase</small>. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: For the remainder of this guide, we will assume you’ve installed the BPS into a folder named <small>biocase</small>, which should look like this by now: ==Running the Setup Script== The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (‘’’biocase’’’) and type in python setup.py (in case Python is not found, you’ll have to add the full path name to the Python executable). Make sure the Python version given at the beginning out the output is the one you desire (otherwise add the path to the correct executable): All of the questions do have defaults. If you want to keep them, just press Enter. Import configuration from existing installation. Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. Webserver domain: This is the domain name of the web server you’re installing BioCASe on, e.g. http//www.foobar.org. You can keep the default localhost as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. Base URL: This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at www.foobar.org/biocase, for example. After answering these three questions, the script will update your files. At the end of the output it will print the lines you will need later to create the link, so do not close the window! ==Linking BioCASe and Your Web Server== The public part of the provider software is stored in the www subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so will compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in C:\biocase, the folder that must be published with a Virtual directory is C:\biocase\www and will be accessible later through http://www.foobar.org/biocase. If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file httpd.conf. This can typically be found in the conf folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! Save the file and restart Apache for the changes to take effect. If you use Internet Information Server, you need to create a virtual directory for the folder c:\biocase\www using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the “Configure” button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add index.html to the list of default documents (virtual directory > properties > documents > default). We know that configuring Python for IIS can be cumbersome; however, there are several good tutorials on the web, for example http://www.ecp.cc/pyiis.html http://support.microsoft.com/default.aspx?scid=kb%3Ben-us%3B276494 http://python.markrowsoft.com/iiswse.asp. Once you’re done, you can test your BioCASe Installation by opening a browser and typing in http://localhost/biocase (replace localhost with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: ==Installing Additional Python Packages== Before you can start configuring your installation, you need to install additional packages, depending on the DBMS used by the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on ‘Utilities’, then “library test” in order to navigate to he library test page that shows the status of installed libraries. On a vanilla BPS installation, it should look similar to this: The BioCASe libraries ship with the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. ==Changing the Default Password== The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking “Start” at the top of the page, or by retyping http://localhost/biocase in the browser window), choose “Config Tool”, then “System Administration”. You will then be prompted for a password. Type in the default ACDC, then click “Submit”. The top of the configuration page you’ll be directed to should look similar to this: In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults localhost and biocase). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press “update config” to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled “Graphviz dot binary”. On Linux machines it might look similar to the default (/usr/local/bin/dot), on Windows machines it will be probably something like C:\Program Files\Graphviz2.26.3\bin\dot.exe. Press “update config” to save the path. ==Checking Your Installation== Before you proceed to the configuration, you should make sure everything is installed correctly: 1. Typing in http://localhost/biocase (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown here) 2. The libraries test page (Utilities/library test) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). 3. “Status of writeable directories” shows a green writeable for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: 9bbaf56a3ad7c7411d6ef44ca27e33bddd81bdc8 19 15 2011-06-21T15:08:58Z JoergHoletschek 3 wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the ___Introduction___ and the ___Beginner’s Guide___ first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e.g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the ___FAQs___ or ___get in contact with the BioCASE Team___. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder <small>biocase</small>. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|200px]][[image:instTortoiseDialog.png|500px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named <small>biocase</small>, which should look like this by now: [[image:instBiocaseFolder.png]] ==Running the Setup Script== The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (‘’’biocase’’’) and type in python setup.py (in case Python is not found, you’ll have to add the full path name to the Python executable). Make sure the Python version given at the beginning out the output is the one you desire (otherwise add the path to the correct executable): All of the questions do have defaults. If you want to keep them, just press Enter. Import configuration from existing installation. Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. Webserver domain: This is the domain name of the web server you’re installing BioCASe on, e.g. http//www.foobar.org. You can keep the default localhost as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. Base URL: This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at www.foobar.org/biocase, for example. After answering these three questions, the script will update your files. At the end of the output it will print the lines you will need later to create the link, so do not close the window! ==Linking BioCASe and Your Web Server== The public part of the provider software is stored in the www subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so will compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in C:\biocase, the folder that must be published with a Virtual directory is C:\biocase\www and will be accessible later through http://www.foobar.org/biocase. If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file httpd.conf. This can typically be found in the conf folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! Save the file and restart Apache for the changes to take effect. If you use Internet Information Server, you need to create a virtual directory for the folder c:\biocase\www using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the “Configure” button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add index.html to the list of default documents (virtual directory > properties > documents > default). We know that configuring Python for IIS can be cumbersome; however, there are several good tutorials on the web, for example http://www.ecp.cc/pyiis.html http://support.microsoft.com/default.aspx?scid=kb%3Ben-us%3B276494 http://python.markrowsoft.com/iiswse.asp. Once you’re done, you can test your BioCASe Installation by opening a browser and typing in http://localhost/biocase (replace localhost with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: ==Installing Additional Python Packages== Before you can start configuring your installation, you need to install additional packages, depending on the DBMS used by the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on ‘Utilities’, then “library test” in order to navigate to he library test page that shows the status of installed libraries. On a vanilla BPS installation, it should look similar to this: The BioCASe libraries ship with the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. ==Changing the Default Password== The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking “Start” at the top of the page, or by retyping http://localhost/biocase in the browser window), choose “Config Tool”, then “System Administration”. You will then be prompted for a password. Type in the default ACDC, then click “Submit”. The top of the configuration page you’ll be directed to should look similar to this: In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults localhost and biocase). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press “update config” to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled “Graphviz dot binary”. On Linux machines it might look similar to the default (/usr/local/bin/dot), on Windows machines it will be probably something like C:\Program Files\Graphviz2.26.3\bin\dot.exe. Press “update config” to save the path. ==Checking Your Installation== Before you proceed to the configuration, you should make sure everything is installed correctly: 1. Typing in http://localhost/biocase (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown here) 2. The libraries test page (Utilities/library test) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). 3. “Status of writeable directories” shows a green writeable for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: 6489696300e0be6495e5dafd813096aed15b4c9c 20 19 2011-06-21T15:20:27Z JoergHoletschek 3 wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the ___Introduction___ and the ___Beginner’s Guide___ first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e.g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the ___FAQs___ or ___get in contact with the BioCASE Team___. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|200px]][[image:instTortoiseDialog.png|500px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in (in case Python is not found, you’ll have to add the full path name to the Python executable):<pre>python setup.py</pre>Make sure the Python version given at the beginning out the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png]] All of the questions do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Import configuration from existing installation:</strong> Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.foobar.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.foobar.org/biocase'', for example. After answering these three questions, the script will update your files. At the end of the output it will print the lines you will need later to create the link, so do not close the window! [[image:instSetupScriptDone.png]] ==Linking BioCASe and Your Web Server== The public part of the provider software is stored in the www subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so will compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in C:\biocase, the folder that must be published with a Virtual directory is C:\biocase\www and will be accessible later through http://www.foobar.org/biocase. If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file httpd.conf. This can typically be found in the conf folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! Save the file and restart Apache for the changes to take effect. If you use Internet Information Server, you need to create a virtual directory for the folder c:\biocase\www using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the “Configure” button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add index.html to the list of default documents (virtual directory > properties > documents > default). We know that configuring Python for IIS can be cumbersome; however, there are several good tutorials on the web, for example http://www.ecp.cc/pyiis.html http://support.microsoft.com/default.aspx?scid=kb%3Ben-us%3B276494 http://python.markrowsoft.com/iiswse.asp. Once you’re done, you can test your BioCASe Installation by opening a browser and typing in http://localhost/biocase (replace localhost with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: ==Installing Additional Python Packages== Before you can start configuring your installation, you need to install additional packages, depending on the DBMS used by the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on ‘Utilities’, then “library test” in order to navigate to he library test page that shows the status of installed libraries. On a vanilla BPS installation, it should look similar to this: The BioCASe libraries ship with the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. ==Changing the Default Password== The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking “Start” at the top of the page, or by retyping http://localhost/biocase in the browser window), choose “Config Tool”, then “System Administration”. You will then be prompted for a password. Type in the default ACDC, then click “Submit”. The top of the configuration page you’ll be directed to should look similar to this: In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults localhost and biocase). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press “update config” to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled “Graphviz dot binary”. On Linux machines it might look similar to the default (/usr/local/bin/dot), on Windows machines it will be probably something like C:\Program Files\Graphviz2.26.3\bin\dot.exe. Press “update config” to save the path. ==Checking Your Installation== Before you proceed to the configuration, you should make sure everything is installed correctly: 1. Typing in http://localhost/biocase (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown here) 2. The libraries test page (Utilities/library test) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). 3. “Status of writeable directories” shows a green writeable for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: a46d18f24a01df6ecbc7d1da4153d61695d7fe66 23 20 2011-06-21T15:24:59Z JoergHoletschek 3 /* Linking BioCASe and Your Web Server */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the ___Introduction___ and the ___Beginner’s Guide___ first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e.g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the ___FAQs___ or ___get in contact with the BioCASE Team___. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|200px]][[image:instTortoiseDialog.png|500px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in (in case Python is not found, you’ll have to add the full path name to the Python executable):<pre>python setup.py</pre>Make sure the Python version given at the beginning out the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png]] All of the questions do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Import configuration from existing installation:</strong> Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.foobar.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.foobar.org/biocase'', for example. After answering these three questions, the script will update your files. At the end of the output it will print the lines you will need later to create the link, so do not close the window! [[image:instSetupScriptDone.png]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so will compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.foobar.org/biocase''. If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[Image:instHttpdConf.png]] Save the file and restart Apache for the changes to take effect. If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can be cumbersome; however, there are several good tutorials on the web, for example http://www.ecp.cc/pyiis.html http://support.microsoft.com/default.aspx?scid=kb%3Ben-us%3B276494 http://python.markrowsoft.com/iiswse.asp. Once you’re done, you can test your BioCASe Installation by opening a browser and typing in http://localhost/biocase (replace localhost with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: ==Installing Additional Python Packages== Before you can start configuring your installation, you need to install additional packages, depending on the DBMS used by the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on ‘Utilities’, then “library test” in order to navigate to he library test page that shows the status of installed libraries. On a vanilla BPS installation, it should look similar to this: The BioCASe libraries ship with the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. ==Changing the Default Password== The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking “Start” at the top of the page, or by retyping http://localhost/biocase in the browser window), choose “Config Tool”, then “System Administration”. You will then be prompted for a password. Type in the default ACDC, then click “Submit”. The top of the configuration page you’ll be directed to should look similar to this: In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults localhost and biocase). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press “update config” to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled “Graphviz dot binary”. On Linux machines it might look similar to the default (/usr/local/bin/dot), on Windows machines it will be probably something like C:\Program Files\Graphviz2.26.3\bin\dot.exe. Press “update config” to save the path. ==Checking Your Installation== Before you proceed to the configuration, you should make sure everything is installed correctly: 1. Typing in http://localhost/biocase (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown here) 2. The libraries test page (Utilities/library test) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). 3. “Status of writeable directories” shows a green writeable for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: b2eff89d58f9bbb23f82a606e5f6adf143ffe2d2 24 23 2011-06-21T15:25:32Z JoergHoletschek 3 /* Linking BioCASe and Your Web Server */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the ___Introduction___ and the ___Beginner’s Guide___ first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e.g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the ___FAQs___ or ___get in contact with the BioCASE Team___. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|200px]][[image:instTortoiseDialog.png|500px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in (in case Python is not found, you’ll have to add the full path name to the Python executable):<pre>python setup.py</pre>Make sure the Python version given at the beginning out the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png]] All of the questions do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Import configuration from existing installation:</strong> Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.foobar.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.foobar.org/biocase'', for example. After answering these three questions, the script will update your files. At the end of the output it will print the lines you will need later to create the link, so do not close the window! [[image:instSetupScriptDone.png]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so will compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.foobar.org/biocase''. If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png]] Save the file and restart Apache for the changes to take effect. If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can be cumbersome; however, there are several good tutorials on the web, for example http://www.ecp.cc/pyiis.html http://support.microsoft.com/default.aspx?scid=kb%3Ben-us%3B276494 http://python.markrowsoft.com/iiswse.asp. Once you’re done, you can test your BioCASe Installation by opening a browser and typing in http://localhost/biocase (replace localhost with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: ==Installing Additional Python Packages== Before you can start configuring your installation, you need to install additional packages, depending on the DBMS used by the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on ‘Utilities’, then “library test” in order to navigate to he library test page that shows the status of installed libraries. On a vanilla BPS installation, it should look similar to this: The BioCASe libraries ship with the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. ==Changing the Default Password== The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking “Start” at the top of the page, or by retyping http://localhost/biocase in the browser window), choose “Config Tool”, then “System Administration”. You will then be prompted for a password. Type in the default ACDC, then click “Submit”. The top of the configuration page you’ll be directed to should look similar to this: In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults localhost and biocase). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press “update config” to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled “Graphviz dot binary”. On Linux machines it might look similar to the default (/usr/local/bin/dot), on Windows machines it will be probably something like C:\Program Files\Graphviz2.26.3\bin\dot.exe. Press “update config” to save the path. ==Checking Your Installation== Before you proceed to the configuration, you should make sure everything is installed correctly: 1. Typing in http://localhost/biocase (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown here) 2. The libraries test page (Utilities/library test) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). 3. “Status of writeable directories” shows a green writeable for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: 9d404df319636d3eccf5a1a8c889e5bcb5c57a1e 25 24 2011-06-21T15:57:22Z JoergHoletschek 3 /* Linking BioCASe and Your Web Server */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the ___Introduction___ and the ___Beginner’s Guide___ first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e.g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the ___FAQs___ or ___get in contact with the BioCASE Team___. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|200px]][[image:instTortoiseDialog.png|500px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in (in case Python is not found, you’ll have to add the full path name to the Python executable):<pre>python setup.py</pre>Make sure the Python version given at the beginning out the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png]] All of the questions do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Import configuration from existing installation:</strong> Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.foobar.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.foobar.org/biocase'', for example. After answering these three questions, the script will update your files. At the end of the output it will print the lines you will need later to create the link, so do not close the window! [[image:instSetupScriptDone.png]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so will compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.foobar.org/biocase''. If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png]] Save the file and restart Apache for the changes to take effect. If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can be cumbersome; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png]] ==Installing Additional Python Packages== Before you can start configuring your installation, you need to install additional packages, depending on the DBMS used by the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on ‘Utilities’, then “library test” in order to navigate to he library test page that shows the status of installed libraries. On a vanilla BPS installation, it should look similar to this: The BioCASe libraries ship with the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. ==Changing the Default Password== The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking “Start” at the top of the page, or by retyping http://localhost/biocase in the browser window), choose “Config Tool”, then “System Administration”. You will then be prompted for a password. Type in the default ACDC, then click “Submit”. The top of the configuration page you’ll be directed to should look similar to this: In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults localhost and biocase). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press “update config” to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled “Graphviz dot binary”. On Linux machines it might look similar to the default (/usr/local/bin/dot), on Windows machines it will be probably something like C:\Program Files\Graphviz2.26.3\bin\dot.exe. Press “update config” to save the path. ==Checking Your Installation== Before you proceed to the configuration, you should make sure everything is installed correctly: 1. Typing in http://localhost/biocase (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown here) 2. The libraries test page (Utilities/library test) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). 3. “Status of writeable directories” shows a green writeable for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: f53f950c7320a9c6b1d9e2e50f90b0dbc5ce3455 28 25 2011-06-21T15:59:50Z JoergHoletschek 3 /* Installing Additional Python Packages */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the ___Introduction___ and the ___Beginner’s Guide___ first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e.g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the ___FAQs___ or ___get in contact with the BioCASE Team___. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|200px]][[image:instTortoiseDialog.png|500px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in (in case Python is not found, you’ll have to add the full path name to the Python executable):<pre>python setup.py</pre>Make sure the Python version given at the beginning out the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png]] All of the questions do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Import configuration from existing installation:</strong> Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.foobar.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.foobar.org/biocase'', for example. After answering these three questions, the script will update your files. At the end of the output it will print the lines you will need later to create the link, so do not close the window! [[image:instSetupScriptDone.png]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so will compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.foobar.org/biocase''. If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png]] Save the file and restart Apache for the changes to take effect. If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can be cumbersome; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the DBMS used by the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to he library test page that shows the status of installed libraries. On a vanilla BPS installation, it should look similar to this: [[image:instLibarariesVanilla.png]] The BioCASe libraries ship with the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. ==Changing the Default Password== The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking “Start” at the top of the page, or by retyping http://localhost/biocase in the browser window), choose “Config Tool”, then “System Administration”. You will then be prompted for a password. Type in the default ACDC, then click “Submit”. The top of the configuration page you’ll be directed to should look similar to this: In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults localhost and biocase). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press “update config” to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled “Graphviz dot binary”. On Linux machines it might look similar to the default (/usr/local/bin/dot), on Windows machines it will be probably something like C:\Program Files\Graphviz2.26.3\bin\dot.exe. Press “update config” to save the path. ==Checking Your Installation== Before you proceed to the configuration, you should make sure everything is installed correctly: 1. Typing in http://localhost/biocase (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown here) 2. The libraries test page (Utilities/library test) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). 3. “Status of writeable directories” shows a green writeable for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: d7d9c4a3f3bf5e57f801a3161246fd794c1a8627 29 28 2011-06-21T16:00:18Z JoergHoletschek 3 /* Installing Additional Python Packages */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the ___Introduction___ and the ___Beginner’s Guide___ first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e.g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the ___FAQs___ or ___get in contact with the BioCASE Team___. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|200px]][[image:instTortoiseDialog.png|500px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in (in case Python is not found, you’ll have to add the full path name to the Python executable):<pre>python setup.py</pre>Make sure the Python version given at the beginning out the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png]] All of the questions do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Import configuration from existing installation:</strong> Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.foobar.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.foobar.org/biocase'', for example. After answering these three questions, the script will update your files. At the end of the output it will print the lines you will need later to create the link, so do not close the window! [[image:instSetupScriptDone.png]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so will compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.foobar.org/biocase''. If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png]] Save the file and restart Apache for the changes to take effect. If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can be cumbersome; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the DBMS used by the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to he library test page that shows the status of installed libraries. On a vanilla BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png]] The BioCASe libraries ship with the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. ==Changing the Default Password== The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking “Start” at the top of the page, or by retyping http://localhost/biocase in the browser window), choose “Config Tool”, then “System Administration”. You will then be prompted for a password. Type in the default ACDC, then click “Submit”. The top of the configuration page you’ll be directed to should look similar to this: In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults localhost and biocase). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press “update config” to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled “Graphviz dot binary”. On Linux machines it might look similar to the default (/usr/local/bin/dot), on Windows machines it will be probably something like C:\Program Files\Graphviz2.26.3\bin\dot.exe. Press “update config” to save the path. ==Checking Your Installation== Before you proceed to the configuration, you should make sure everything is installed correctly: 1. Typing in http://localhost/biocase (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown here) 2. The libraries test page (Utilities/library test) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). 3. “Status of writeable directories” shows a green writeable for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: ee56e8617c9ac0f0e3c2e089b70de62ab2234e8a 31 29 2011-06-21T16:05:27Z JoergHoletschek 3 /* Changing the Default Password */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the ___Introduction___ and the ___Beginner’s Guide___ first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e.g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the ___FAQs___ or ___get in contact with the BioCASE Team___. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|200px]][[image:instTortoiseDialog.png|500px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in (in case Python is not found, you’ll have to add the full path name to the Python executable):<pre>python setup.py</pre>Make sure the Python version given at the beginning out the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png]] All of the questions do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Import configuration from existing installation:</strong> Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.foobar.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.foobar.org/biocase'', for example. After answering these three questions, the script will update your files. At the end of the output it will print the lines you will need later to create the link, so do not close the window! [[image:instSetupScriptDone.png]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so will compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.foobar.org/biocase''. If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png]] Save the file and restart Apache for the changes to take effect. If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can be cumbersome; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the DBMS used by the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to he library test page that shows the status of installed libraries. On a vanilla BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png]] The BioCASe libraries ship with the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' in the browser window), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ==Checking Your Installation== Before you proceed to the configuration, you should make sure everything is installed correctly: 1. Typing in http://localhost/biocase (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown here) 2. The libraries test page (Utilities/library test) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). 3. “Status of writeable directories” shows a green writeable for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: f79d36718e1bb9e5bb3c4d6b87514d58026533e8 33 31 2011-06-21T16:08:44Z JoergHoletschek 3 /* Checking Your Installation */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the ___Introduction___ and the ___Beginner’s Guide___ first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e.g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the ___FAQs___ or ___get in contact with the BioCASE Team___. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|200px]][[image:instTortoiseDialog.png|500px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in (in case Python is not found, you’ll have to add the full path name to the Python executable):<pre>python setup.py</pre>Make sure the Python version given at the beginning out the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png]] All of the questions do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Import configuration from existing installation:</strong> Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.foobar.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.foobar.org/biocase'', for example. After answering these three questions, the script will update your files. At the end of the output it will print the lines you will need later to create the link, so do not close the window! [[image:instSetupScriptDone.png]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so will compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.foobar.org/biocase''. If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png]] Save the file and restart Apache for the changes to take effect. If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can be cumbersome; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the DBMS used by the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to he library test page that shows the status of installed libraries. On a vanilla BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png]] The BioCASe libraries ship with the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' in the browser window), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown here) # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png]] 0b0570f9deed2eb23c252c7a140733b8b9de021f 35 33 2011-06-21T16:13:58Z JoergHoletschek 3 /* Checking Your Installation */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the ___Introduction___ and the ___Beginner’s Guide___ first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e.g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the ___FAQs___ or ___get in contact with the BioCASE Team___. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|200px]][[image:instTortoiseDialog.png|500px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in (in case Python is not found, you’ll have to add the full path name to the Python executable):<pre>python setup.py</pre>Make sure the Python version given at the beginning out the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png]] All of the questions do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Import configuration from existing installation:</strong> Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.foobar.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.foobar.org/biocase'', for example. After answering these three questions, the script will update your files. At the end of the output it will print the lines you will need later to create the link, so do not close the window! [[image:instSetupScriptDone.png]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so will compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.foobar.org/biocase''. If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png]] Save the file and restart Apache for the changes to take effect. If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can be cumbersome; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the DBMS used by the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to he library test page that shows the status of installed libraries. On a vanilla BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png]] The BioCASe libraries ship with the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' in the browser window), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown [[ Installation ]] ). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png]] b1bcdcc6311b51d9c34b45dc4970b083597b53cb 36 35 2011-06-21T16:15:42Z JoergHoletschek 3 /* Checking Your Installation */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the ___Introduction___ and the ___Beginner’s Guide___ first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e.g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the ___FAQs___ or ___get in contact with the BioCASE Team___. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|200px]][[image:instTortoiseDialog.png|500px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in (in case Python is not found, you’ll have to add the full path name to the Python executable):<pre>python setup.py</pre>Make sure the Python version given at the beginning out the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png]] All of the questions do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Import configuration from existing installation:</strong> Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.foobar.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.foobar.org/biocase'', for example. After answering these three questions, the script will update your files. At the end of the output it will print the lines you will need later to create the link, so do not close the window! [[image:instSetupScriptDone.png]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so will compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.foobar.org/biocase''. If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png]] Save the file and restart Apache for the changes to take effect. If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can be cumbersome; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the DBMS used by the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to he library test page that shows the status of installed libraries. On a vanilla BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png]] The BioCASe libraries ship with the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' in the browser window), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown [[Installation#Linking_BioCASe_and_Your_Web_Server|here]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png]] 4a6780b7f6a390662633a17f0480b11cefb3b532 37 36 2011-06-21T16:19:22Z JoergHoletschek 3 /* Downloading the Provider Software */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the ___Introduction___ and the ___Beginner’s Guide___ first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e.g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the ___FAQs___ or ___get in contact with the BioCASE Team___. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in (in case Python is not found, you’ll have to add the full path name to the Python executable):<pre>python setup.py</pre>Make sure the Python version given at the beginning out the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png]] All of the questions do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Import configuration from existing installation:</strong> Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.foobar.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.foobar.org/biocase'', for example. After answering these three questions, the script will update your files. At the end of the output it will print the lines you will need later to create the link, so do not close the window! [[image:instSetupScriptDone.png]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so will compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.foobar.org/biocase''. If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png]] Save the file and restart Apache for the changes to take effect. If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can be cumbersome; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the DBMS used by the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to he library test page that shows the status of installed libraries. On a vanilla BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png]] The BioCASe libraries ship with the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' in the browser window), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown [[Installation#Linking_BioCASe_and_Your_Web_Server|here]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png]] 72a2f6fa0a618d05c6b7a2ce571ff795f9cd94e2 38 37 2011-06-21T16:23:43Z JoergHoletschek 3 wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the ___Introduction___ and the ___Beginner’s Guide___ first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e.g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the ___FAQs___ or ___get in contact with the BioCASE Team___. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in (in case Python is not found, you’ll have to add the full path name to the Python executable):<pre>python setup.py</pre>Make sure the Python version given at the beginning out the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] All of the questions do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Import configuration from existing installation:</strong> Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.foobar.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.foobar.org/biocase'', for example. After answering these three questions, the script will update your files. At the end of the output it will print the lines you will need later to create the link, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so will compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.foobar.org/biocase''. If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can be cumbersome; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the DBMS used by the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to he library test page that shows the status of installed libraries. On a vanilla BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|480px]] The BioCASe libraries ship with the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' in the browser window), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown [[Installation#Linking_BioCASe_and_Your_Web_Server|here]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|480px]] adf7ebb9c180b026c0c1ea5edc01d96f78ae339e 39 38 2011-06-21T16:25:19Z JoergHoletschek 3 wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the ___Introduction___ and the ___Beginner’s Guide___ first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e.g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the ___FAQs___ or ___get in contact with the BioCASE Team___. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in (in case Python is not found, you’ll have to add the full path name to the Python executable):<pre>python setup.py</pre>Make sure the Python version given at the beginning out the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] All of the questions do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Import configuration from existing installation:</strong> Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.foobar.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.foobar.org/biocase'', for example. After answering these three questions, the script will update your files. At the end of the output it will print the lines you will need later to create the link, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so will compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.foobar.org/biocase''. If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can be cumbersome; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the DBMS used by the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to he library test page that shows the status of installed libraries. On a vanilla BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries ship with the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' in the browser window), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown [[Installation#Linking_BioCASe_and_Your_Web_Server|here]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] e689c5962822e7974e2ecfb8f98b61dde1a2a70b 40 39 2011-06-21T16:29:37Z JoergHoletschek 3 /* Linking BioCASe and Your Web Server */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the ___Introduction___ and the ___Beginner’s Guide___ first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e.g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the ___FAQs___ or ___get in contact with the BioCASE Team___. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in (in case Python is not found, you’ll have to add the full path name to the Python executable):<pre>python setup.py</pre>Make sure the Python version given at the beginning out the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] All of the questions do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Import configuration from existing installation:</strong> Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.foobar.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.foobar.org/biocase'', for example. After answering these three questions, the script will update your files. At the end of the output it will print the lines you will need later to create the link, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so will compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.foobar.org/biocase''. If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can a pain; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the DBMS used by the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to he library test page that shows the status of installed libraries. On a vanilla BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries ship with the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' in the browser window), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown [[Installation#Linking_BioCASe_and_Your_Web_Server|here]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] 3b06f27efe976ed91570fe46d88a8eaadc34fdf3 6 7 16 2011-06-21T14:58:14Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 8 17 2011-06-21T15:05:34Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 9 18 2011-06-21T15:05:52Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 10 21 2011-06-21T15:20:57Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 11 22 2011-06-21T15:21:20Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 12 26 2011-06-21T15:57:33Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 13 27 2011-06-21T15:57:49Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 14 30 2011-06-21T16:00:56Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 15 32 2011-06-21T16:05:36Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 16 34 2011-06-21T16:08:54Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 0 17 41 2011-06-22T12:57:59Z JoergHoletschek 3 Created page with "Before you start publishing your database with BioCASe, you need to consider several issues. In some cases, some preparation of your database for the publication might be necessa..." wikitext text/x-wiki Before you start publishing your database with BioCASe, you need to consider several issues. In some cases, some preparation of your database for the publication might be necessary. You should read this page of the wiki careful before you start configuring the Provider Software for a given database. Live-DB versus Snapshot The easiest way is to connect BioCASe to the same database you’re working with. This would result in any changes you do being visible on the BioCASe access point right away – which is an idea you might not like. Some curators prefer to publish defined versions of their database, for example after they’ve finished reviewing all specimens of a certain genus. If you prefer this, you need to create a second database that holds defined snapshots of your working database and that will be published instead. Snapshots could be generated in a regular interval (e.g., each Friday evening) or upon request. Please contact your IT guys for more help on this. Access Control The BioCASe Provider Software will only publish your data to biodiversity networks, meaning it will only read from the database and never do any changes. For security reasons it’s a good idea to create a user for BioCASe that has only read privileges on the database – more precisely on the tables you want to publish. Since the configuration tool will try to load a list of tables and columns from the database to allow easier setup, this user should also be allowed to load meta-information (SHOW TABLES, SHOW COLUMNS). Metadata Metadata are information that describe your dataset as a whole - title, description, curator’s name, ownership, date of last modification, terms of use etc. BioCASe can publish a lot of metadata – and you should take the time to provide as much information as you can, since this will increase the usability of your published data. For storing metadata, please create an extra table in your database and fill it in. If you go to the BioCASe folder of your installation, you’ll find the template for this table in sampledb\abcdmetadata. Either use the SQL script to create it in your database, or import it from the Access .mdb. Per default, all records of your database will make up one dataset, being described by one set of metadata. In case your database holds records of several datasets, you can also group these into several datasets in ABCD by providing different sets of metadata. So when you realize during gathering of meta-information that they’re not the same for all records, this is an indicator that your database must be grouped into several datasets. If all records to be published share the same metadata, the metadata table needs to have just one row. If they don’t, it needs to hold several rows, one set of metadata for each dataset, identified by a unique primary key (1, 2, …). In order to associate each record of your database with the correct set of metadata, you will need to add a foreign key into the base record table that links to the respective entry in the metadata table. Ask your IT guys if you need help on that. Controlled Denormalisation As you will see during configuration, the Provider Software is able to join together the different tables of your database’s data model in order to retrieve all information to be published from the database. This works good if the number of tables is within reasonable limits, that is up to approximately 10-15. Data models used by off-the-self collection management system are usually highly normalised and use far more tables. Specify’s data model, for example, has about 70 tables. Configuring such a complex model in BioCASe will be a terrible pain; moreover, in case your database holds a huge number of records, the time required by the Provider Software for answering search request will become too long. Therefore it is advisable in such cases to add some controlled denormalisation to your database, that is to export the data to be published into several cache tables. These tables could be stored in the same database as the source tables or separate in an extra cache database, maybe even on a different server. They could be created in a regular interval (e.g. each night) or upon request by SQL scripts. The figure below shows this denormalisation process for the data model of Specify. On the left side, you see some of Specify’s source tables (only the ones starting with the letters A, B, C and D). With an SQL script the will be reduced to 7 cache tables, one of them being abcd_object. These tables hold only the columns that will be published, drawn from up to 15 source tables. Repeatable Elements in ABCD One of the main features of ABCD (the schema you will probably use to publish your data) is repeatable elements. For example, ABCD can store multiple identifications or several URLs for multimedia objects attached to a specimen record. All these URLs will be stored in the same ABCD element – namely MultimediaObject/FileURI. Another example is the higher taxonomy (all taxa above genus level). Typically these are stored in different columns in databases (see figure below). In ABCD, they will all end up in the same element, HigherTaxon/Name. In order to be able to map the different ranks stored in columns to the same ABCD concept, we need to pivot these columns into rows – each rank kept in a column needs to be turned into a row, with two columns storing taxon name and rank, one column for linking to the according specimen record, plus one newly created primary key: Such a transformation can be easily done with a database view. Views do exactly what the name suggests – they provide a different view on the data. They can be used like regular tables, but they merely represent a different representation of another table. Updates of the underlying tables (or insert/deletions) will also affect views created on these tables, since the data is stored only once in the database. Let’s assume you have the higher taxonomy stored in the specimen table like this: The SQL statement CREATE VIEW [dbo].[vwHigherTaxa] AS SELECT 'k_' + [EDIT_ATBI_RecordID] AS id, [EDIT_ATBI_RecordID] AS unit_id, [kingdom] AS name, 'kingdom' AS rank FROM unit_data WHERE [kingdom] IS NOT NULL UNION SELECT 'p_' + [EDIT_ATBI_RecordID], [EDIT_ATBI_RecordID], [phylum], 'phylum‚ FROM unit_data WHERE [phylum] IS NOT NULL UNION ... Will create a view that meets the requirements described above. Remember that even though it looks like a table, it’s just another representation of the data stored in the table unit_data. Such pivoting must be done in all cases when several columns need to be mapped to the same (repeatable) ABCD element. If it’s just one value to be mapped, there is no need for that. Commonly used ABCD elements that are repeatable and might ask for such preparatory transformations are - Identification - HigherTaxon - GatheringSite/NamedArea - Metadata/Scope/GeoecologicalTerms - Metadata/Scope/TaxonomicTerms - MultimediaObjects - MeasurementsOrFacts If you have problems preparing your database for publication, please contact the ___Biocase Support___. 80eddef81ba5b899a850cd000372e91ee2ae28e7 42 41 2011-06-22T13:03:32Z JoergHoletschek 3 wikitext text/x-wiki Before you start publishing your database with BioCASe, you need to consider several issues. In some cases, some preparation of your database for the publication might be necessary. You should read this page of the wiki careful before you start configuring the Provider Software for a given database. == Live-DB versus Snapshot == The easiest way is to connect BioCASe to the same database you’re working with. This would result in any changes you do being visible on the BioCASe access point right away – which is an idea you might not like. Some curators prefer to publish defined versions of their database, for example after they’ve finished reviewing all specimens of a certain genus. If you prefer this, you need to create a second database that holds defined snapshots of your working database and that will be published instead. Snapshots could be generated in a regular interval (e.g., each Friday evening) or upon request. Please contact your IT guys for more help on this. == Access Control == The BioCASe Provider Software will only publish your data to biodiversity networks, meaning it will only read from the database and never do any changes. For security reasons it’s a good idea to create a user for BioCASe that has only read privileges on the database – more precisely on the tables you want to publish. Since the configuration tool will try to load a list of tables and columns from the database to allow easier setup, this user should also be allowed to load meta-information (SHOW TABLES, SHOW COLUMNS). == Metadata == Metadata are information that describe your dataset as a whole - title, description, curator’s name, ownership, date of last modification, terms of use etc. BioCASe can publish a lot of metadata – and you should take the time to provide as much information as you can, since this will increase the usability of your published data. For storing metadata, please create an extra table in your database and fill it in. If you go to the BioCASe folder of your installation, you’ll find the template for this table in sampledb\abcdmetadata. Either use the SQL script to create it in your database, or import it from the Access .mdb. Per default, all records of your database will make up one dataset, being described by one set of metadata. In case your database holds records of several datasets, you can also group these into several datasets in ABCD by providing different sets of metadata. So when you realize during gathering of meta-information that they’re not the same for all records, this is an indicator that your database must be grouped into several datasets. If all records to be published share the same metadata, the metadata table needs to have just one row. If they don’t, it needs to hold several rows, one set of metadata for each dataset, identified by a unique primary key (1, 2, …). In order to associate each record of your database with the correct set of metadata, you will need to add a foreign key into the base record table that links to the respective entry in the metadata table. Ask your IT guys if you need help on that. == Controlled Denormalisation == As you will see during configuration, the Provider Software is able to join together the different tables of your database’s data model in order to retrieve all information to be published from the database. This works good if the number of tables is within reasonable limits, that is up to approximately 10-15. Data models used by off-the-self collection management system are usually highly normalised and use far more tables. Specify’s data model, for example, has about 70 tables. Configuring such a complex model in BioCASe will be a terrible pain; moreover, in case your database holds a huge number of records, the time required by the Provider Software for answering search request will become too long. Therefore it is advisable in such cases to add some controlled denormalisation to your database, that is to export the data to be published into several cache tables. These tables could be stored in the same database as the source tables or separate in an extra cache database, maybe even on a different server. They could be created in a regular interval (e.g. each night) or upon request by SQL scripts. The figure below shows this denormalisation process for the data model of Specify. On the left side, you see some of Specify’s source tables (only the ones starting with the letters A, B, C and D). With an SQL script the will be reduced to 7 cache tables, one of them being abcd_object. These tables hold only the columns that will be published, drawn from up to 15 source tables. == Repeatable Elements in ABCD == One of the main features of ABCD (the schema you will probably use to publish your data) is repeatable elements. For example, ABCD can store multiple identifications or several URLs for multimedia objects attached to a specimen record. All these URLs will be stored in the same ABCD element – namely MultimediaObject/FileURI. Another example is the higher taxonomy (all taxa above genus level). Typically these are stored in different columns in databases (see figure below). In ABCD, they will all end up in the same element, HigherTaxon/Name. In order to be able to map the different ranks stored in columns to the same ABCD concept, we need to pivot these columns into rows – each rank kept in a column needs to be turned into a row, with two columns storing taxon name and rank, one column for linking to the according specimen record, plus one newly created primary key: Such a transformation can be easily done with a database view. Views do exactly what the name suggests – they provide a different view on the data. They can be used like regular tables, but they merely represent a different representation of another table. Updates of the underlying tables (or insert/deletions) will also affect views created on these tables, since the data is stored only once in the database. Let’s assume you have the higher taxonomy stored in the specimen table like this: The SQL statement CREATE VIEW [dbo].[vwHigherTaxa] AS SELECT 'k_' + [EDIT_ATBI_RecordID] AS id, [EDIT_ATBI_RecordID] AS unit_id, [kingdom] AS name, 'kingdom' AS rank FROM unit_data WHERE [kingdom] IS NOT NULL UNION SELECT 'p_' + [EDIT_ATBI_RecordID], [EDIT_ATBI_RecordID], [phylum], 'phylum‚ FROM unit_data WHERE [phylum] IS NOT NULL UNION ... Will create a view that meets the requirements described above. Remember that even though it looks like a table, it’s just another representation of the data stored in the table unit_data. Such pivoting must be done in all cases when several columns need to be mapped to the same (repeatable) ABCD element. If it’s just one value to be mapped, there is no need for that. Commonly used ABCD elements that are repeatable and might ask for such preparatory transformations are - Identification - HigherTaxon - GatheringSite/NamedArea - Metadata/Scope/GeoecologicalTerms - Metadata/Scope/TaxonomicTerms - MultimediaObjects - MeasurementsOrFacts If you have problems preparing your database for publication, please contact the ___Biocase Support___. 5a738bd0745c019aab1156f154f5ebef13b13126 43 42 2011-06-22T13:04:16Z JoergHoletschek 3 wikitext text/x-wiki = Preparation of your database = Before you start publishing your database with BioCASe, you need to consider several issues. In some cases, some preparation of your database for the publication might be necessary. You should read this page of the wiki careful before you start configuring the Provider Software for a given database. == Live-DB versus Snapshot == The easiest way is to connect BioCASe to the same database you’re working with. This would result in any changes you do being visible on the BioCASe access point right away – which is an idea you might not like. Some curators prefer to publish defined versions of their database, for example after they’ve finished reviewing all specimens of a certain genus. If you prefer this, you need to create a second database that holds defined snapshots of your working database and that will be published instead. Snapshots could be generated in a regular interval (e.g., each Friday evening) or upon request. Please contact your IT guys for more help on this. == Access Control == The BioCASe Provider Software will only publish your data to biodiversity networks, meaning it will only read from the database and never do any changes. For security reasons it’s a good idea to create a user for BioCASe that has only read privileges on the database – more precisely on the tables you want to publish. Since the configuration tool will try to load a list of tables and columns from the database to allow easier setup, this user should also be allowed to load meta-information (SHOW TABLES, SHOW COLUMNS). == Metadata == Metadata are information that describe your dataset as a whole - title, description, curator’s name, ownership, date of last modification, terms of use etc. BioCASe can publish a lot of metadata – and you should take the time to provide as much information as you can, since this will increase the usability of your published data. For storing metadata, please create an extra table in your database and fill it in. If you go to the BioCASe folder of your installation, you’ll find the template for this table in sampledb\abcdmetadata. Either use the SQL script to create it in your database, or import it from the Access .mdb. Per default, all records of your database will make up one dataset, being described by one set of metadata. In case your database holds records of several datasets, you can also group these into several datasets in ABCD by providing different sets of metadata. So when you realize during gathering of meta-information that they’re not the same for all records, this is an indicator that your database must be grouped into several datasets. If all records to be published share the same metadata, the metadata table needs to have just one row. If they don’t, it needs to hold several rows, one set of metadata for each dataset, identified by a unique primary key (1, 2, …). In order to associate each record of your database with the correct set of metadata, you will need to add a foreign key into the base record table that links to the respective entry in the metadata table. Ask your IT guys if you need help on that. == Controlled Denormalisation == As you will see during configuration, the Provider Software is able to join together the different tables of your database’s data model in order to retrieve all information to be published from the database. This works good if the number of tables is within reasonable limits, that is up to approximately 10-15. Data models used by off-the-self collection management system are usually highly normalised and use far more tables. Specify’s data model, for example, has about 70 tables. Configuring such a complex model in BioCASe will be a terrible pain; moreover, in case your database holds a huge number of records, the time required by the Provider Software for answering search request will become too long. Therefore it is advisable in such cases to add some controlled denormalisation to your database, that is to export the data to be published into several cache tables. These tables could be stored in the same database as the source tables or separate in an extra cache database, maybe even on a different server. They could be created in a regular interval (e.g. each night) or upon request by SQL scripts. The figure below shows this denormalisation process for the data model of Specify. On the left side, you see some of Specify’s source tables (only the ones starting with the letters A, B, C and D). With an SQL script the will be reduced to 7 cache tables, one of them being abcd_object. These tables hold only the columns that will be published, drawn from up to 15 source tables. == Repeatable Elements in ABCD == One of the main features of ABCD (the schema you will probably use to publish your data) is repeatable elements. For example, ABCD can store multiple identifications or several URLs for multimedia objects attached to a specimen record. All these URLs will be stored in the same ABCD element – namely MultimediaObject/FileURI. Another example is the higher taxonomy (all taxa above genus level). Typically these are stored in different columns in databases (see figure below). In ABCD, they will all end up in the same element, HigherTaxon/Name. In order to be able to map the different ranks stored in columns to the same ABCD concept, we need to pivot these columns into rows – each rank kept in a column needs to be turned into a row, with two columns storing taxon name and rank, one column for linking to the according specimen record, plus one newly created primary key: Such a transformation can be easily done with a database view. Views do exactly what the name suggests – they provide a different view on the data. They can be used like regular tables, but they merely represent a different representation of another table. Updates of the underlying tables (or insert/deletions) will also affect views created on these tables, since the data is stored only once in the database. Let’s assume you have the higher taxonomy stored in the specimen table like this: The SQL statement CREATE VIEW [dbo].[vwHigherTaxa] AS SELECT 'k_' + [EDIT_ATBI_RecordID] AS id, [EDIT_ATBI_RecordID] AS unit_id, [kingdom] AS name, 'kingdom' AS rank FROM unit_data WHERE [kingdom] IS NOT NULL UNION SELECT 'p_' + [EDIT_ATBI_RecordID], [EDIT_ATBI_RecordID], [phylum], 'phylum‚ FROM unit_data WHERE [phylum] IS NOT NULL UNION ... Will create a view that meets the requirements described above. Remember that even though it looks like a table, it’s just another representation of the data stored in the table unit_data. Such pivoting must be done in all cases when several columns need to be mapped to the same (repeatable) ABCD element. If it’s just one value to be mapped, there is no need for that. Commonly used ABCD elements that are repeatable and might ask for such preparatory transformations are - Identification - HigherTaxon - GatheringSite/NamedArea - Metadata/Scope/GeoecologicalTerms - Metadata/Scope/TaxonomicTerms - MultimediaObjects - MeasurementsOrFacts If you have problems preparing your database for publication, please contact the ___Biocase Support___. 95985b0562ad37ef4736b7b87acd49be74d61d16 44 43 2011-06-22T13:06:32Z JoergHoletschek 3 wikitext text/x-wiki Preparation of your database Before you start publishing your database with BioCASe, you need to consider several issues. In some cases, some preparation of your database for the publication might be necessary. You should read this page of the wiki careful before you start configuring the Provider Software for a given database. == Live-DB versus Snapshot == The easiest way is to connect BioCASe to the same database you’re working with. This would result in any changes you do being visible on the BioCASe access point right away – which is an idea you might not like. Some curators prefer to publish defined versions of their database, for example after they’ve finished reviewing all specimens of a certain genus. If you prefer this, you need to create a second database that holds defined snapshots of your working database and that will be published instead. Snapshots could be generated in a regular interval (e.g., each Friday evening) or upon request. Please contact your IT guys for more help on this. == Access Control == The BioCASe Provider Software will only publish your data to biodiversity networks, meaning it will only read from the database and never do any changes. For security reasons it’s a good idea to create a user for BioCASe that has only read privileges on the database – more precisely on the tables you want to publish. Since the configuration tool will try to load a list of tables and columns from the database to allow easier setup, this user should also be allowed to load meta-information (SHOW TABLES, SHOW COLUMNS). == Metadata == Metadata are information that describe your dataset as a whole - title, description, curator’s name, ownership, date of last modification, terms of use etc. BioCASe can publish a lot of metadata – and you should take the time to provide as much information as you can, since this will increase the usability of your published data. For storing metadata, please create an extra table in your database and fill it in. If you go to the BioCASe folder of your installation, you’ll find the template for this table in sampledb\abcdmetadata. Either use the SQL script to create it in your database, or import it from the Access .mdb. Per default, all records of your database will make up one dataset, being described by one set of metadata. In case your database holds records of several datasets, you can also group these into several datasets in ABCD by providing different sets of metadata. So when you realize during gathering of meta-information that they’re not the same for all records, this is an indicator that your database must be grouped into several datasets. If all records to be published share the same metadata, the metadata table needs to have just one row. If they don’t, it needs to hold several rows, one set of metadata for each dataset, identified by a unique primary key (1, 2, …). In order to associate each record of your database with the correct set of metadata, you will need to add a foreign key into the base record table that links to the respective entry in the metadata table. Ask your IT guys if you need help on that. == Controlled Denormalisation == As you will see during configuration, the Provider Software is able to join together the different tables of your database’s data model in order to retrieve all information to be published from the database. This works good if the number of tables is within reasonable limits, that is up to approximately 10-15. Data models used by off-the-self collection management system are usually highly normalised and use far more tables. Specify’s data model, for example, has about 70 tables. Configuring such a complex model in BioCASe will be a terrible pain; moreover, in case your database holds a huge number of records, the time required by the Provider Software for answering search request will become too long. Therefore it is advisable in such cases to add some controlled denormalisation to your database, that is to export the data to be published into several cache tables. These tables could be stored in the same database as the source tables or separate in an extra cache database, maybe even on a different server. They could be created in a regular interval (e.g. each night) or upon request by SQL scripts. The figure below shows this denormalisation process for the data model of Specify. On the left side, you see some of Specify’s source tables (only the ones starting with the letters A, B, C and D). With an SQL script the will be reduced to 7 cache tables, one of them being abcd_object. These tables hold only the columns that will be published, drawn from up to 15 source tables. == Repeatable Elements in ABCD == One of the main features of ABCD (the schema you will probably use to publish your data) is repeatable elements. For example, ABCD can store multiple identifications or several URLs for multimedia objects attached to a specimen record. All these URLs will be stored in the same ABCD element – namely MultimediaObject/FileURI. Another example is the higher taxonomy (all taxa above genus level). Typically these are stored in different columns in databases (see figure below). In ABCD, they will all end up in the same element, HigherTaxon/Name. In order to be able to map the different ranks stored in columns to the same ABCD concept, we need to pivot these columns into rows – each rank kept in a column needs to be turned into a row, with two columns storing taxon name and rank, one column for linking to the according specimen record, plus one newly created primary key: Such a transformation can be easily done with a database view. Views do exactly what the name suggests – they provide a different view on the data. They can be used like regular tables, but they merely represent a different representation of another table. Updates of the underlying tables (or insert/deletions) will also affect views created on these tables, since the data is stored only once in the database. Let’s assume you have the higher taxonomy stored in the specimen table like this: The SQL statement CREATE VIEW [dbo].[vwHigherTaxa] AS SELECT 'k_' + [EDIT_ATBI_RecordID] AS id, [EDIT_ATBI_RecordID] AS unit_id, [kingdom] AS name, 'kingdom' AS rank FROM unit_data WHERE [kingdom] IS NOT NULL UNION SELECT 'p_' + [EDIT_ATBI_RecordID], [EDIT_ATBI_RecordID], [phylum], 'phylum‚ FROM unit_data WHERE [phylum] IS NOT NULL UNION ... Will create a view that meets the requirements described above. Remember that even though it looks like a table, it’s just another representation of the data stored in the table unit_data. Such pivoting must be done in all cases when several columns need to be mapped to the same (repeatable) ABCD element. If it’s just one value to be mapped, there is no need for that. Commonly used ABCD elements that are repeatable and might ask for such preparatory transformations are - Identification - HigherTaxon - GatheringSite/NamedArea - Metadata/Scope/GeoecologicalTerms - Metadata/Scope/TaxonomicTerms - MultimediaObjects - MeasurementsOrFacts If you have problems preparing your database for publication, please contact the ___Biocase Support___. 0cccfefe66847ff9f923d0fb4a46fa3adcc81b84 45 44 2011-06-22T13:06:52Z JoergHoletschek 3 wikitext text/x-wiki Before you start publishing your database with BioCASe, you need to consider several issues. In some cases, some preparation of your database for the publication might be necessary. You should read this page of the wiki careful before you start configuring the Provider Software for a given database. == Live-DB versus Snapshot == The easiest way is to connect BioCASe to the same database you’re working with. This would result in any changes you do being visible on the BioCASe access point right away – which is an idea you might not like. Some curators prefer to publish defined versions of their database, for example after they’ve finished reviewing all specimens of a certain genus. If you prefer this, you need to create a second database that holds defined snapshots of your working database and that will be published instead. Snapshots could be generated in a regular interval (e.g., each Friday evening) or upon request. Please contact your IT guys for more help on this. == Access Control == The BioCASe Provider Software will only publish your data to biodiversity networks, meaning it will only read from the database and never do any changes. For security reasons it’s a good idea to create a user for BioCASe that has only read privileges on the database – more precisely on the tables you want to publish. Since the configuration tool will try to load a list of tables and columns from the database to allow easier setup, this user should also be allowed to load meta-information (SHOW TABLES, SHOW COLUMNS). == Metadata == Metadata are information that describe your dataset as a whole - title, description, curator’s name, ownership, date of last modification, terms of use etc. BioCASe can publish a lot of metadata – and you should take the time to provide as much information as you can, since this will increase the usability of your published data. For storing metadata, please create an extra table in your database and fill it in. If you go to the BioCASe folder of your installation, you’ll find the template for this table in sampledb\abcdmetadata. Either use the SQL script to create it in your database, or import it from the Access .mdb. Per default, all records of your database will make up one dataset, being described by one set of metadata. In case your database holds records of several datasets, you can also group these into several datasets in ABCD by providing different sets of metadata. So when you realize during gathering of meta-information that they’re not the same for all records, this is an indicator that your database must be grouped into several datasets. If all records to be published share the same metadata, the metadata table needs to have just one row. If they don’t, it needs to hold several rows, one set of metadata for each dataset, identified by a unique primary key (1, 2, …). In order to associate each record of your database with the correct set of metadata, you will need to add a foreign key into the base record table that links to the respective entry in the metadata table. Ask your IT guys if you need help on that. == Controlled Denormalisation == As you will see during configuration, the Provider Software is able to join together the different tables of your database’s data model in order to retrieve all information to be published from the database. This works good if the number of tables is within reasonable limits, that is up to approximately 10-15. Data models used by off-the-self collection management system are usually highly normalised and use far more tables. Specify’s data model, for example, has about 70 tables. Configuring such a complex model in BioCASe will be a terrible pain; moreover, in case your database holds a huge number of records, the time required by the Provider Software for answering search request will become too long. Therefore it is advisable in such cases to add some controlled denormalisation to your database, that is to export the data to be published into several cache tables. These tables could be stored in the same database as the source tables or separate in an extra cache database, maybe even on a different server. They could be created in a regular interval (e.g. each night) or upon request by SQL scripts. The figure below shows this denormalisation process for the data model of Specify. On the left side, you see some of Specify’s source tables (only the ones starting with the letters A, B, C and D). With an SQL script the will be reduced to 7 cache tables, one of them being abcd_object. These tables hold only the columns that will be published, drawn from up to 15 source tables. == Repeatable Elements in ABCD == One of the main features of ABCD (the schema you will probably use to publish your data) is repeatable elements. For example, ABCD can store multiple identifications or several URLs for multimedia objects attached to a specimen record. All these URLs will be stored in the same ABCD element – namely MultimediaObject/FileURI. Another example is the higher taxonomy (all taxa above genus level). Typically these are stored in different columns in databases (see figure below). In ABCD, they will all end up in the same element, HigherTaxon/Name. In order to be able to map the different ranks stored in columns to the same ABCD concept, we need to pivot these columns into rows – each rank kept in a column needs to be turned into a row, with two columns storing taxon name and rank, one column for linking to the according specimen record, plus one newly created primary key: Such a transformation can be easily done with a database view. Views do exactly what the name suggests – they provide a different view on the data. They can be used like regular tables, but they merely represent a different representation of another table. Updates of the underlying tables (or insert/deletions) will also affect views created on these tables, since the data is stored only once in the database. Let’s assume you have the higher taxonomy stored in the specimen table like this: The SQL statement CREATE VIEW [dbo].[vwHigherTaxa] AS SELECT 'k_' + [EDIT_ATBI_RecordID] AS id, [EDIT_ATBI_RecordID] AS unit_id, [kingdom] AS name, 'kingdom' AS rank FROM unit_data WHERE [kingdom] IS NOT NULL UNION SELECT 'p_' + [EDIT_ATBI_RecordID], [EDIT_ATBI_RecordID], [phylum], 'phylum‚ FROM unit_data WHERE [phylum] IS NOT NULL UNION ... Will create a view that meets the requirements described above. Remember that even though it looks like a table, it’s just another representation of the data stored in the table unit_data. Such pivoting must be done in all cases when several columns need to be mapped to the same (repeatable) ABCD element. If it’s just one value to be mapped, there is no need for that. Commonly used ABCD elements that are repeatable and might ask for such preparatory transformations are - Identification - HigherTaxon - GatheringSite/NamedArea - Metadata/Scope/GeoecologicalTerms - Metadata/Scope/TaxonomicTerms - MultimediaObjects - MeasurementsOrFacts If you have problems preparing your database for publication, please contact the ___Biocase Support___. 5a738bd0745c019aab1156f154f5ebef13b13126 46 45 2011-06-22T13:07:46Z JoergHoletschek 3 /* Access Control */ wikitext text/x-wiki Before you start publishing your database with BioCASe, you need to consider several issues. In some cases, some preparation of your database for the publication might be necessary. You should read this page of the wiki careful before you start configuring the Provider Software for a given database. == Live-DB versus Snapshot == The easiest way is to connect BioCASe to the same database you’re working with. This would result in any changes you do being visible on the BioCASe access point right away – which is an idea you might not like. Some curators prefer to publish defined versions of their database, for example after they’ve finished reviewing all specimens of a certain genus. If you prefer this, you need to create a second database that holds defined snapshots of your working database and that will be published instead. Snapshots could be generated in a regular interval (e.g., each Friday evening) or upon request. Please contact your IT guys for more help on this. == Access Control == The BioCASe Provider Software will only publish your data to biodiversity networks, meaning it will only read from the database and never do any changes. For security reasons it’s a good idea to create a user for BioCASe that has only read privileges on the database – more precisely on the tables you want to publish. Since the configuration tool will try to load a list of tables and columns from the database to allow easier setup, this user should also be allowed to load meta-information (<small>SHOW TABLES, SHOW COLUMNS</small>). == Metadata == Metadata are information that describe your dataset as a whole - title, description, curator’s name, ownership, date of last modification, terms of use etc. BioCASe can publish a lot of metadata – and you should take the time to provide as much information as you can, since this will increase the usability of your published data. For storing metadata, please create an extra table in your database and fill it in. If you go to the BioCASe folder of your installation, you’ll find the template for this table in sampledb\abcdmetadata. Either use the SQL script to create it in your database, or import it from the Access .mdb. Per default, all records of your database will make up one dataset, being described by one set of metadata. In case your database holds records of several datasets, you can also group these into several datasets in ABCD by providing different sets of metadata. So when you realize during gathering of meta-information that they’re not the same for all records, this is an indicator that your database must be grouped into several datasets. If all records to be published share the same metadata, the metadata table needs to have just one row. If they don’t, it needs to hold several rows, one set of metadata for each dataset, identified by a unique primary key (1, 2, …). In order to associate each record of your database with the correct set of metadata, you will need to add a foreign key into the base record table that links to the respective entry in the metadata table. Ask your IT guys if you need help on that. == Controlled Denormalisation == As you will see during configuration, the Provider Software is able to join together the different tables of your database’s data model in order to retrieve all information to be published from the database. This works good if the number of tables is within reasonable limits, that is up to approximately 10-15. Data models used by off-the-self collection management system are usually highly normalised and use far more tables. Specify’s data model, for example, has about 70 tables. Configuring such a complex model in BioCASe will be a terrible pain; moreover, in case your database holds a huge number of records, the time required by the Provider Software for answering search request will become too long. Therefore it is advisable in such cases to add some controlled denormalisation to your database, that is to export the data to be published into several cache tables. These tables could be stored in the same database as the source tables or separate in an extra cache database, maybe even on a different server. They could be created in a regular interval (e.g. each night) or upon request by SQL scripts. The figure below shows this denormalisation process for the data model of Specify. On the left side, you see some of Specify’s source tables (only the ones starting with the letters A, B, C and D). With an SQL script the will be reduced to 7 cache tables, one of them being abcd_object. These tables hold only the columns that will be published, drawn from up to 15 source tables. == Repeatable Elements in ABCD == One of the main features of ABCD (the schema you will probably use to publish your data) is repeatable elements. For example, ABCD can store multiple identifications or several URLs for multimedia objects attached to a specimen record. All these URLs will be stored in the same ABCD element – namely MultimediaObject/FileURI. Another example is the higher taxonomy (all taxa above genus level). Typically these are stored in different columns in databases (see figure below). In ABCD, they will all end up in the same element, HigherTaxon/Name. In order to be able to map the different ranks stored in columns to the same ABCD concept, we need to pivot these columns into rows – each rank kept in a column needs to be turned into a row, with two columns storing taxon name and rank, one column for linking to the according specimen record, plus one newly created primary key: Such a transformation can be easily done with a database view. Views do exactly what the name suggests – they provide a different view on the data. They can be used like regular tables, but they merely represent a different representation of another table. Updates of the underlying tables (or insert/deletions) will also affect views created on these tables, since the data is stored only once in the database. Let’s assume you have the higher taxonomy stored in the specimen table like this: The SQL statement CREATE VIEW [dbo].[vwHigherTaxa] AS SELECT 'k_' + [EDIT_ATBI_RecordID] AS id, [EDIT_ATBI_RecordID] AS unit_id, [kingdom] AS name, 'kingdom' AS rank FROM unit_data WHERE [kingdom] IS NOT NULL UNION SELECT 'p_' + [EDIT_ATBI_RecordID], [EDIT_ATBI_RecordID], [phylum], 'phylum‚ FROM unit_data WHERE [phylum] IS NOT NULL UNION ... Will create a view that meets the requirements described above. Remember that even though it looks like a table, it’s just another representation of the data stored in the table unit_data. Such pivoting must be done in all cases when several columns need to be mapped to the same (repeatable) ABCD element. If it’s just one value to be mapped, there is no need for that. Commonly used ABCD elements that are repeatable and might ask for such preparatory transformations are - Identification - HigherTaxon - GatheringSite/NamedArea - Metadata/Scope/GeoecologicalTerms - Metadata/Scope/TaxonomicTerms - MultimediaObjects - MeasurementsOrFacts If you have problems preparing your database for publication, please contact the ___Biocase Support___. 0467ae0183ec8ae59a76cf48c2c16c4b289d3ec4 47 46 2011-06-22T13:09:39Z JoergHoletschek 3 /* Metadata */ wikitext text/x-wiki Before you start publishing your database with BioCASe, you need to consider several issues. In some cases, some preparation of your database for the publication might be necessary. You should read this page of the wiki careful before you start configuring the Provider Software for a given database. == Live-DB versus Snapshot == The easiest way is to connect BioCASe to the same database you’re working with. This would result in any changes you do being visible on the BioCASe access point right away – which is an idea you might not like. Some curators prefer to publish defined versions of their database, for example after they’ve finished reviewing all specimens of a certain genus. If you prefer this, you need to create a second database that holds defined snapshots of your working database and that will be published instead. Snapshots could be generated in a regular interval (e.g., each Friday evening) or upon request. Please contact your IT guys for more help on this. == Access Control == The BioCASe Provider Software will only publish your data to biodiversity networks, meaning it will only read from the database and never do any changes. For security reasons it’s a good idea to create a user for BioCASe that has only read privileges on the database – more precisely on the tables you want to publish. Since the configuration tool will try to load a list of tables and columns from the database to allow easier setup, this user should also be allowed to load meta-information (<small>SHOW TABLES, SHOW COLUMNS</small>). == Metadata == Metadata are information that describe your dataset as a whole - title, description, curator’s name, ownership, date of last modification, terms of use etc. BioCASe can publish a lot of metadata – and you should take the time to provide as much information as you can, since this will increase the usability of your published data. For storing metadata, please create an extra table in your database and fill it in. If you go to the BioCASe folder of your installation, you’ll find the template for this table in ''sampledb/abcdmetadata''. Either use the SQL script to create it in your database, or import it from the Access database (''metadata.mdb''). Per default, all records of your database will make up one dataset, being described by one set of metadata. In case your database holds records of several datasets, you can also group these into several datasets in ABCD by providing different sets of metadata. So when you realize during gathering of meta-information that they’re not the same for all records, this is an indicator that your database must be grouped into several datasets. If all records to be published share the same metadata, the metadata table needs to have just one row. If they don’t, it needs to hold several rows, one set of metadata for each dataset, identified by a unique primary key (1, 2, …). In order to associate each record of your database with the correct set of metadata, you will need to add a foreign key into the base record table that links to the respective entry in the metadata table. Ask your IT guys if you need help on that. == Controlled Denormalisation == As you will see during configuration, the Provider Software is able to join together the different tables of your database’s data model in order to retrieve all information to be published from the database. This works good if the number of tables is within reasonable limits, that is up to approximately 10-15. Data models used by off-the-self collection management system are usually highly normalised and use far more tables. Specify’s data model, for example, has about 70 tables. Configuring such a complex model in BioCASe will be a terrible pain; moreover, in case your database holds a huge number of records, the time required by the Provider Software for answering search request will become too long. Therefore it is advisable in such cases to add some controlled denormalisation to your database, that is to export the data to be published into several cache tables. These tables could be stored in the same database as the source tables or separate in an extra cache database, maybe even on a different server. They could be created in a regular interval (e.g. each night) or upon request by SQL scripts. The figure below shows this denormalisation process for the data model of Specify. On the left side, you see some of Specify’s source tables (only the ones starting with the letters A, B, C and D). With an SQL script the will be reduced to 7 cache tables, one of them being abcd_object. These tables hold only the columns that will be published, drawn from up to 15 source tables. == Repeatable Elements in ABCD == One of the main features of ABCD (the schema you will probably use to publish your data) is repeatable elements. For example, ABCD can store multiple identifications or several URLs for multimedia objects attached to a specimen record. All these URLs will be stored in the same ABCD element – namely MultimediaObject/FileURI. Another example is the higher taxonomy (all taxa above genus level). Typically these are stored in different columns in databases (see figure below). In ABCD, they will all end up in the same element, HigherTaxon/Name. In order to be able to map the different ranks stored in columns to the same ABCD concept, we need to pivot these columns into rows – each rank kept in a column needs to be turned into a row, with two columns storing taxon name and rank, one column for linking to the according specimen record, plus one newly created primary key: Such a transformation can be easily done with a database view. Views do exactly what the name suggests – they provide a different view on the data. They can be used like regular tables, but they merely represent a different representation of another table. Updates of the underlying tables (or insert/deletions) will also affect views created on these tables, since the data is stored only once in the database. Let’s assume you have the higher taxonomy stored in the specimen table like this: The SQL statement CREATE VIEW [dbo].[vwHigherTaxa] AS SELECT 'k_' + [EDIT_ATBI_RecordID] AS id, [EDIT_ATBI_RecordID] AS unit_id, [kingdom] AS name, 'kingdom' AS rank FROM unit_data WHERE [kingdom] IS NOT NULL UNION SELECT 'p_' + [EDIT_ATBI_RecordID], [EDIT_ATBI_RecordID], [phylum], 'phylum‚ FROM unit_data WHERE [phylum] IS NOT NULL UNION ... Will create a view that meets the requirements described above. Remember that even though it looks like a table, it’s just another representation of the data stored in the table unit_data. Such pivoting must be done in all cases when several columns need to be mapped to the same (repeatable) ABCD element. If it’s just one value to be mapped, there is no need for that. Commonly used ABCD elements that are repeatable and might ask for such preparatory transformations are - Identification - HigherTaxon - GatheringSite/NamedArea - Metadata/Scope/GeoecologicalTerms - Metadata/Scope/TaxonomicTerms - MultimediaObjects - MeasurementsOrFacts If you have problems preparing your database for publication, please contact the ___Biocase Support___. b10124223d3f22ce7e90fd34807050bc60a49530 48 47 2011-06-22T13:10:52Z JoergHoletschek 3 /* Controlled Denormalisation */ wikitext text/x-wiki Before you start publishing your database with BioCASe, you need to consider several issues. In some cases, some preparation of your database for the publication might be necessary. You should read this page of the wiki careful before you start configuring the Provider Software for a given database. == Live-DB versus Snapshot == The easiest way is to connect BioCASe to the same database you’re working with. This would result in any changes you do being visible on the BioCASe access point right away – which is an idea you might not like. Some curators prefer to publish defined versions of their database, for example after they’ve finished reviewing all specimens of a certain genus. If you prefer this, you need to create a second database that holds defined snapshots of your working database and that will be published instead. Snapshots could be generated in a regular interval (e.g., each Friday evening) or upon request. Please contact your IT guys for more help on this. == Access Control == The BioCASe Provider Software will only publish your data to biodiversity networks, meaning it will only read from the database and never do any changes. For security reasons it’s a good idea to create a user for BioCASe that has only read privileges on the database – more precisely on the tables you want to publish. Since the configuration tool will try to load a list of tables and columns from the database to allow easier setup, this user should also be allowed to load meta-information (<small>SHOW TABLES, SHOW COLUMNS</small>). == Metadata == Metadata are information that describe your dataset as a whole - title, description, curator’s name, ownership, date of last modification, terms of use etc. BioCASe can publish a lot of metadata – and you should take the time to provide as much information as you can, since this will increase the usability of your published data. For storing metadata, please create an extra table in your database and fill it in. If you go to the BioCASe folder of your installation, you’ll find the template for this table in ''sampledb/abcdmetadata''. Either use the SQL script to create it in your database, or import it from the Access database (''metadata.mdb''). Per default, all records of your database will make up one dataset, being described by one set of metadata. In case your database holds records of several datasets, you can also group these into several datasets in ABCD by providing different sets of metadata. So when you realize during gathering of meta-information that they’re not the same for all records, this is an indicator that your database must be grouped into several datasets. If all records to be published share the same metadata, the metadata table needs to have just one row. If they don’t, it needs to hold several rows, one set of metadata for each dataset, identified by a unique primary key (1, 2, …). In order to associate each record of your database with the correct set of metadata, you will need to add a foreign key into the base record table that links to the respective entry in the metadata table. Ask your IT guys if you need help on that. == Controlled Denormalisation == As you will see during configuration, the Provider Software is able to join together the different tables of your database’s data model in order to retrieve all information to be published from the database. This works good if the number of tables is within reasonable limits, that is up to approximately 10-15. Data models used by off-the-self collection management system are usually highly normalised and use far more tables. Specify’s data model, for example, has about 70 tables. Configuring such a complex model in BioCASe will be a terrible pain; moreover, in case your database holds a huge number of records, the time required by the Provider Software for answering search request will become too long. Therefore it is advisable in such cases to add some controlled denormalisation to your database, that is to export the data to be published into several cache tables. These tables could be stored in the same database as the source tables or separate in an extra cache database, maybe even on a different server. They could be created in a regular interval (e.g. each night) or upon request by SQL scripts. The figure below shows this denormalisation process for the data model of Specify. On the left side, you see some of Specify’s source tables (only the ones starting with the letters A, B, C and D). With an SQL script the will be reduced to 7 cache tables, one of them being abcd_object. These tables hold only the columns that will be published, drawn from up to 15 source tables. [[image:prepDenormalisation.png]] == Repeatable Elements in ABCD == One of the main features of ABCD (the schema you will probably use to publish your data) is repeatable elements. For example, ABCD can store multiple identifications or several URLs for multimedia objects attached to a specimen record. All these URLs will be stored in the same ABCD element – namely MultimediaObject/FileURI. Another example is the higher taxonomy (all taxa above genus level). Typically these are stored in different columns in databases (see figure below). In ABCD, they will all end up in the same element, HigherTaxon/Name. In order to be able to map the different ranks stored in columns to the same ABCD concept, we need to pivot these columns into rows – each rank kept in a column needs to be turned into a row, with two columns storing taxon name and rank, one column for linking to the according specimen record, plus one newly created primary key: Such a transformation can be easily done with a database view. Views do exactly what the name suggests – they provide a different view on the data. They can be used like regular tables, but they merely represent a different representation of another table. Updates of the underlying tables (or insert/deletions) will also affect views created on these tables, since the data is stored only once in the database. Let’s assume you have the higher taxonomy stored in the specimen table like this: The SQL statement CREATE VIEW [dbo].[vwHigherTaxa] AS SELECT 'k_' + [EDIT_ATBI_RecordID] AS id, [EDIT_ATBI_RecordID] AS unit_id, [kingdom] AS name, 'kingdom' AS rank FROM unit_data WHERE [kingdom] IS NOT NULL UNION SELECT 'p_' + [EDIT_ATBI_RecordID], [EDIT_ATBI_RecordID], [phylum], 'phylum‚ FROM unit_data WHERE [phylum] IS NOT NULL UNION ... Will create a view that meets the requirements described above. Remember that even though it looks like a table, it’s just another representation of the data stored in the table unit_data. Such pivoting must be done in all cases when several columns need to be mapped to the same (repeatable) ABCD element. If it’s just one value to be mapped, there is no need for that. Commonly used ABCD elements that are repeatable and might ask for such preparatory transformations are - Identification - HigherTaxon - GatheringSite/NamedArea - Metadata/Scope/GeoecologicalTerms - Metadata/Scope/TaxonomicTerms - MultimediaObjects - MeasurementsOrFacts If you have problems preparing your database for publication, please contact the ___Biocase Support___. f8ba32099416fbdfd42b1fd09cdaf9bd3eb24ca7 50 48 2011-06-22T13:11:52Z JoergHoletschek 3 /* Controlled Denormalisation */ wikitext text/x-wiki Before you start publishing your database with BioCASe, you need to consider several issues. In some cases, some preparation of your database for the publication might be necessary. You should read this page of the wiki careful before you start configuring the Provider Software for a given database. == Live-DB versus Snapshot == The easiest way is to connect BioCASe to the same database you’re working with. This would result in any changes you do being visible on the BioCASe access point right away – which is an idea you might not like. Some curators prefer to publish defined versions of their database, for example after they’ve finished reviewing all specimens of a certain genus. If you prefer this, you need to create a second database that holds defined snapshots of your working database and that will be published instead. Snapshots could be generated in a regular interval (e.g., each Friday evening) or upon request. Please contact your IT guys for more help on this. == Access Control == The BioCASe Provider Software will only publish your data to biodiversity networks, meaning it will only read from the database and never do any changes. For security reasons it’s a good idea to create a user for BioCASe that has only read privileges on the database – more precisely on the tables you want to publish. Since the configuration tool will try to load a list of tables and columns from the database to allow easier setup, this user should also be allowed to load meta-information (<small>SHOW TABLES, SHOW COLUMNS</small>). == Metadata == Metadata are information that describe your dataset as a whole - title, description, curator’s name, ownership, date of last modification, terms of use etc. BioCASe can publish a lot of metadata – and you should take the time to provide as much information as you can, since this will increase the usability of your published data. For storing metadata, please create an extra table in your database and fill it in. If you go to the BioCASe folder of your installation, you’ll find the template for this table in ''sampledb/abcdmetadata''. Either use the SQL script to create it in your database, or import it from the Access database (''metadata.mdb''). Per default, all records of your database will make up one dataset, being described by one set of metadata. In case your database holds records of several datasets, you can also group these into several datasets in ABCD by providing different sets of metadata. So when you realize during gathering of meta-information that they’re not the same for all records, this is an indicator that your database must be grouped into several datasets. If all records to be published share the same metadata, the metadata table needs to have just one row. If they don’t, it needs to hold several rows, one set of metadata for each dataset, identified by a unique primary key (1, 2, …). In order to associate each record of your database with the correct set of metadata, you will need to add a foreign key into the base record table that links to the respective entry in the metadata table. Ask your IT guys if you need help on that. == Controlled Denormalisation == As you will see during configuration, the Provider Software is able to join together the different tables of your database’s data model in order to retrieve all information to be published from the database. This works good if the number of tables is within reasonable limits, that is up to approximately 10-15. Data models used by off-the-self collection management system are usually highly normalised and use far more tables. Specify’s data model, for example, has about 70 tables. Configuring such a complex model in BioCASe will be a terrible pain; moreover, in case your database holds a huge number of records, the time required by the Provider Software for answering search request will become too long. Therefore it is advisable in such cases to add some controlled denormalisation to your database, that is to export the data to be published into several cache tables. These tables could be stored in the same database as the source tables or separate in an extra cache database, maybe even on a different server. They could be created in a regular interval (e.g. each night) or upon request by SQL scripts. The figure below shows this denormalisation process for the data model of Specify. On the left side, you see some of Specify’s source tables (only the ones starting with the letters A, B, C and D). With an SQL script the will be reduced to 7 cache tables, one of them being abcd_object. These tables hold only the columns that will be published, drawn from up to 15 source tables. [[image:prepDenormalisation.png|750px]] == Repeatable Elements in ABCD == One of the main features of ABCD (the schema you will probably use to publish your data) is repeatable elements. For example, ABCD can store multiple identifications or several URLs for multimedia objects attached to a specimen record. All these URLs will be stored in the same ABCD element – namely MultimediaObject/FileURI. Another example is the higher taxonomy (all taxa above genus level). Typically these are stored in different columns in databases (see figure below). In ABCD, they will all end up in the same element, HigherTaxon/Name. In order to be able to map the different ranks stored in columns to the same ABCD concept, we need to pivot these columns into rows – each rank kept in a column needs to be turned into a row, with two columns storing taxon name and rank, one column for linking to the according specimen record, plus one newly created primary key: Such a transformation can be easily done with a database view. Views do exactly what the name suggests – they provide a different view on the data. They can be used like regular tables, but they merely represent a different representation of another table. Updates of the underlying tables (or insert/deletions) will also affect views created on these tables, since the data is stored only once in the database. Let’s assume you have the higher taxonomy stored in the specimen table like this: The SQL statement CREATE VIEW [dbo].[vwHigherTaxa] AS SELECT 'k_' + [EDIT_ATBI_RecordID] AS id, [EDIT_ATBI_RecordID] AS unit_id, [kingdom] AS name, 'kingdom' AS rank FROM unit_data WHERE [kingdom] IS NOT NULL UNION SELECT 'p_' + [EDIT_ATBI_RecordID], [EDIT_ATBI_RecordID], [phylum], 'phylum‚ FROM unit_data WHERE [phylum] IS NOT NULL UNION ... Will create a view that meets the requirements described above. Remember that even though it looks like a table, it’s just another representation of the data stored in the table unit_data. Such pivoting must be done in all cases when several columns need to be mapped to the same (repeatable) ABCD element. If it’s just one value to be mapped, there is no need for that. Commonly used ABCD elements that are repeatable and might ask for such preparatory transformations are - Identification - HigherTaxon - GatheringSite/NamedArea - Metadata/Scope/GeoecologicalTerms - Metadata/Scope/TaxonomicTerms - MultimediaObjects - MeasurementsOrFacts If you have problems preparing your database for publication, please contact the ___Biocase Support___. e0240c5ab26acebb56f6574bcc1cf77adb2fde03 6 18 49 2011-06-22T13:11:09Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 0 17 51 50 2011-06-22T13:16:31Z JoergHoletschek 3 /* Repeatable Elements in ABCD */ wikitext text/x-wiki Before you start publishing your database with BioCASe, you need to consider several issues. In some cases, some preparation of your database for the publication might be necessary. You should read this page of the wiki careful before you start configuring the Provider Software for a given database. == Live-DB versus Snapshot == The easiest way is to connect BioCASe to the same database you’re working with. This would result in any changes you do being visible on the BioCASe access point right away – which is an idea you might not like. Some curators prefer to publish defined versions of their database, for example after they’ve finished reviewing all specimens of a certain genus. If you prefer this, you need to create a second database that holds defined snapshots of your working database and that will be published instead. Snapshots could be generated in a regular interval (e.g., each Friday evening) or upon request. Please contact your IT guys for more help on this. == Access Control == The BioCASe Provider Software will only publish your data to biodiversity networks, meaning it will only read from the database and never do any changes. For security reasons it’s a good idea to create a user for BioCASe that has only read privileges on the database – more precisely on the tables you want to publish. Since the configuration tool will try to load a list of tables and columns from the database to allow easier setup, this user should also be allowed to load meta-information (<small>SHOW TABLES, SHOW COLUMNS</small>). == Metadata == Metadata are information that describe your dataset as a whole - title, description, curator’s name, ownership, date of last modification, terms of use etc. BioCASe can publish a lot of metadata – and you should take the time to provide as much information as you can, since this will increase the usability of your published data. For storing metadata, please create an extra table in your database and fill it in. If you go to the BioCASe folder of your installation, you’ll find the template for this table in ''sampledb/abcdmetadata''. Either use the SQL script to create it in your database, or import it from the Access database (''metadata.mdb''). Per default, all records of your database will make up one dataset, being described by one set of metadata. In case your database holds records of several datasets, you can also group these into several datasets in ABCD by providing different sets of metadata. So when you realize during gathering of meta-information that they’re not the same for all records, this is an indicator that your database must be grouped into several datasets. If all records to be published share the same metadata, the metadata table needs to have just one row. If they don’t, it needs to hold several rows, one set of metadata for each dataset, identified by a unique primary key (1, 2, …). In order to associate each record of your database with the correct set of metadata, you will need to add a foreign key into the base record table that links to the respective entry in the metadata table. Ask your IT guys if you need help on that. == Controlled Denormalisation == As you will see during configuration, the Provider Software is able to join together the different tables of your database’s data model in order to retrieve all information to be published from the database. This works good if the number of tables is within reasonable limits, that is up to approximately 10-15. Data models used by off-the-self collection management system are usually highly normalised and use far more tables. Specify’s data model, for example, has about 70 tables. Configuring such a complex model in BioCASe will be a terrible pain; moreover, in case your database holds a huge number of records, the time required by the Provider Software for answering search request will become too long. Therefore it is advisable in such cases to add some controlled denormalisation to your database, that is to export the data to be published into several cache tables. These tables could be stored in the same database as the source tables or separate in an extra cache database, maybe even on a different server. They could be created in a regular interval (e.g. each night) or upon request by SQL scripts. The figure below shows this denormalisation process for the data model of Specify. On the left side, you see some of Specify’s source tables (only the ones starting with the letters A, B, C and D). With an SQL script the will be reduced to 7 cache tables, one of them being abcd_object. These tables hold only the columns that will be published, drawn from up to 15 source tables. [[image:prepDenormalisation.png|750px]] == Repeatable Elements in ABCD == One of the main features of ABCD (the schema you will probably use to publish your data) is repeatable elements. For example, ABCD can store multiple identifications or several URLs for multimedia objects attached to a specimen record. All these URLs will be stored in the same ABCD element – namely ''MultimediaObject/FileURI''. Another example is the higher taxonomy (all taxa above genus level). Typically these are stored in different columns in databases (see figure below). In ABCD, they will all end up in the same element, ''HigherTaxon/Name''. In order to be able to map the different ranks stored in columns to the same ABCD concept, we need to pivot these columns into rows – each rank kept in a column needs to be turned into a row, with two columns storing taxon name and rank, one column for linking to the according specimen record, plus one newly created primary key: [[File:prepPivoting.png]] Such a transformation can be easily done with a database view. Views do exactly what the name suggests – they provide a different view on the data. They can be used like regular tables, but they merely represent a different representation of another table. Updates of the underlying tables (or insert/deletions) will also affect views created on these tables, since the data is stored only once in the database. Let’s assume you have the higher taxonomy stored in the specimen table like this: [[File:prepEditTable.png]] The SQL statement <pre> CREATE VIEW [dbo].[vwHigherTaxa] AS SELECT 'k_' + [EDIT_ATBI_RecordID] AS id, [EDIT_ATBI_RecordID] AS unit_id, [kingdom] AS name, 'kingdom' AS rank FROM unit_data WHERE [kingdom] IS NOT NULL UNION ALL SELECT 'p_' + [EDIT_ATBI_RecordID], [EDIT_ATBI_RecordID], [phylum], 'phylum‚ FROM unit_data WHERE [phylum] IS NOT NULL UNION ALL ... </pre> will create a view that meets the requirements described above. Remember that even though it looks like a table, it’s just another representation of the data stored in the table unit_data: [[File:prepEditView.png]] Such pivoting must be done in all cases when several columns need to be mapped to the same (repeatable) ABCD element. If it’s just one value to be mapped, there is no need for that. Commonly used ABCD elements that are repeatable and might ask for such preparatory transformations are * Identification * HigherTaxon * GatheringSite/NamedArea * Metadata/Scope/GeoecologicalTerms * Metadata/Scope/TaxonomicTerms * MultimediaObjects * MeasurementsOrFacts If you have problems preparing your database for publication, please contact the ___Biocase Support___. 41b08128efd2fb591b18d82463d6f0ab2a44a15f 55 51 2011-06-22T13:19:29Z JoergHoletschek 3 /* Repeatable Elements in ABCD */ wikitext text/x-wiki Before you start publishing your database with BioCASe, you need to consider several issues. In some cases, some preparation of your database for the publication might be necessary. You should read this page of the wiki careful before you start configuring the Provider Software for a given database. == Live-DB versus Snapshot == The easiest way is to connect BioCASe to the same database you’re working with. This would result in any changes you do being visible on the BioCASe access point right away – which is an idea you might not like. Some curators prefer to publish defined versions of their database, for example after they’ve finished reviewing all specimens of a certain genus. If you prefer this, you need to create a second database that holds defined snapshots of your working database and that will be published instead. Snapshots could be generated in a regular interval (e.g., each Friday evening) or upon request. Please contact your IT guys for more help on this. == Access Control == The BioCASe Provider Software will only publish your data to biodiversity networks, meaning it will only read from the database and never do any changes. For security reasons it’s a good idea to create a user for BioCASe that has only read privileges on the database – more precisely on the tables you want to publish. Since the configuration tool will try to load a list of tables and columns from the database to allow easier setup, this user should also be allowed to load meta-information (<small>SHOW TABLES, SHOW COLUMNS</small>). == Metadata == Metadata are information that describe your dataset as a whole - title, description, curator’s name, ownership, date of last modification, terms of use etc. BioCASe can publish a lot of metadata – and you should take the time to provide as much information as you can, since this will increase the usability of your published data. For storing metadata, please create an extra table in your database and fill it in. If you go to the BioCASe folder of your installation, you’ll find the template for this table in ''sampledb/abcdmetadata''. Either use the SQL script to create it in your database, or import it from the Access database (''metadata.mdb''). Per default, all records of your database will make up one dataset, being described by one set of metadata. In case your database holds records of several datasets, you can also group these into several datasets in ABCD by providing different sets of metadata. So when you realize during gathering of meta-information that they’re not the same for all records, this is an indicator that your database must be grouped into several datasets. If all records to be published share the same metadata, the metadata table needs to have just one row. If they don’t, it needs to hold several rows, one set of metadata for each dataset, identified by a unique primary key (1, 2, …). In order to associate each record of your database with the correct set of metadata, you will need to add a foreign key into the base record table that links to the respective entry in the metadata table. Ask your IT guys if you need help on that. == Controlled Denormalisation == As you will see during configuration, the Provider Software is able to join together the different tables of your database’s data model in order to retrieve all information to be published from the database. This works good if the number of tables is within reasonable limits, that is up to approximately 10-15. Data models used by off-the-self collection management system are usually highly normalised and use far more tables. Specify’s data model, for example, has about 70 tables. Configuring such a complex model in BioCASe will be a terrible pain; moreover, in case your database holds a huge number of records, the time required by the Provider Software for answering search request will become too long. Therefore it is advisable in such cases to add some controlled denormalisation to your database, that is to export the data to be published into several cache tables. These tables could be stored in the same database as the source tables or separate in an extra cache database, maybe even on a different server. They could be created in a regular interval (e.g. each night) or upon request by SQL scripts. The figure below shows this denormalisation process for the data model of Specify. On the left side, you see some of Specify’s source tables (only the ones starting with the letters A, B, C and D). With an SQL script the will be reduced to 7 cache tables, one of them being abcd_object. These tables hold only the columns that will be published, drawn from up to 15 source tables. [[image:prepDenormalisation.png|750px]] == Repeatable Elements in ABCD == One of the main features of ABCD (the schema you will probably use to publish your data) is repeatable elements. For example, ABCD can store multiple identifications or several URLs for multimedia objects attached to a specimen record. All these URLs will be stored in the same ABCD element – namely ''MultimediaObject/FileURI''. Another example is the higher taxonomy (all taxa above genus level). Typically these are stored in different columns in databases (see figure below). In ABCD, they will all end up in the same element, ''HigherTaxon/Name''. In order to be able to map the different ranks stored in columns to the same ABCD concept, we need to pivot these columns into rows – each rank kept in a column needs to be turned into a row, with two columns storing taxon name and rank, one column for linking to the according specimen record, plus one newly created primary key: [[File:prepPivoting.png]] Such a transformation can be easily done with a database view. Views do exactly what the name suggests – they provide a different view on the data. They can be used like regular tables, but they merely represent a different representation of another table. Updates of the underlying tables (or insert/deletions) will also affect views created on these tables, since the data is stored only once in the database. Let’s assume you have the higher taxonomy stored in the specimen table like this: [[File:prepEditTable.png]] The SQL statement <pre> CREATE VIEW [dbo].[vwHigherTaxa] AS SELECT 'k_' + [EDIT_ATBI_RecordID] AS id, [EDIT_ATBI_RecordID] AS unit_id, [kingdom] AS name, 'kingdom' AS rank FROM unit_data WHERE [kingdom] IS NOT NULL UNION ALL SELECT 'p_' + [EDIT_ATBI_RecordID], [EDIT_ATBI_RecordID], [phylum], 'phylum'‚ FROM unit_data WHERE [phylum] IS NOT NULL UNION ALL ... </pre> will create a view that meets the requirements described above. Remember that even though it looks like a table, it’s just another representation of the data stored in the table unit_data: [[File:prepEditView.png]] Such pivoting must be done in all cases when several columns need to be mapped to the same (repeatable) ABCD element. If it’s just one value to be mapped, there is no need for that. Commonly used ABCD elements that are repeatable and might ask for such preparatory transformations are * Identification * HigherTaxon * GatheringSite/NamedArea * Metadata/Scope/GeoecologicalTerms * Metadata/Scope/TaxonomicTerms * MultimediaObjects * MeasurementsOrFacts If you have problems preparing your database for publication, please contact the ___Biocase Support___. 1fa920693fbf69dc07794c24b220707c18cab553 6 19 52 2011-06-22T13:17:46Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 20 53 2011-06-22T13:18:08Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 21 54 2011-06-22T13:18:27Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 0 22 56 2011-06-22T13:49:20Z JoergHoletschek 3 Created page with "This page lists the most commonly used ABCD 2.06 concepts. The full ABCD documentation can be found here. == Mandatory == List of mandatory concepts. Without them being mapped ..." wikitext text/x-wiki This page lists the most commonly used ABCD 2.06 concepts. The full ABCD documentation can be found here. == Mandatory == List of mandatory concepts. Without them being mapped the Provider Software will not be able to create valid ABCD document: Metadata * /DataSets/DataSet/ContentContacts/ContentContact/Name * /DataSets/DataSet/TechnicalContacts/TechnicalContact/Name * /DataSets/DataSet/Metadata/Description/Representation/Title * /DataSets/DataSet/Metadata/Description/Representation[@language] * /DataSets/DataSet/Metadata/RevisionData/DateModified Unitdata * /DataSets/DataSet/Units/Unit/SourceInstitutionID * /DataSets/DataSet/Units/Unit/SourceID * /DataSets/DataSet/Units/Unit/UnitID == Highly Recommended == List of recommended ABCD concepts. If you can provide this data, please do so, they're commonly used entry points to your data. Metadata * /DataSets/DataSet/ContentContacts/ContentContact/Email * /DataSets/DataSet/ContentContacts/ContentContact/Address * /DataSets/DataSet/TechnicalContacts/TechnicalContact/Email * /DataSets/DataSet/TechnicalContacts/TechnicalContact/Address * /DataSets/DataSet/Metadata/Description/Representation/Details * /DataSets/DataSet/Metadata/Owners/Owner/Organisation/Name/Representation/Text * /DataSets/DataSet/Metadata/Owners/Owner/Person/FullName * /DataSets/DataSet/Metadata/Owners/Owner/URIs/URL * /DataSets/DataSet/Metadata/Owners/Owner/LogoURI Unitdata * /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString * /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/HigherTaxa/HigherTaxon/HigherTaxonName * /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/HigherTaxa/HigherTaxon/HigherTaxonRank * /DataSets/DataSet/Units/Unit/Identifications/Identification/PreferredFlag * /DataSets/DataSet/Units/Unit/RecordBasis * /DataSets/DataSet/Units/Unit/Gathering/DateTime/ISODateTimeBegin * /DataSets/DataSet/Units/Unit/Gathering/Agents/GatheringAgent/Person/FullName * /DataSets/DataSet/Units/Unit/Gathering/LocalityText * /DataSets/DataSet/Units/Unit/Gathering/Country/ISO3166Code * /DataSets/DataSet/Units/Unit/Gathering/Country/Name * /DataSets/DataSet/Units/Unit/Gathering/SiteCoordinateSets/SiteCoordinates/CoordinatesLatLong/LongitudeDecimal */DataSets/DataSet/Units/Unit/Gathering/SiteCoordinateSets/SiteCoordinates/CoordinatesLatLong/LatitudeDecimal == Commonly Used == List of further commonly mapped ABCD concepts. Metadata * /DataSets/DataSet/IPRStatements/TermsOfUseStatements/TermsOfUse/Text * /DataSets/DataSet/IPRStatements/Copyrights/Copyright/Text * /DataSets/DataSet/IPRStatements/IPRDeclarations/IPRDeclaration/Text * /DataSets/DataSet/IPRStatements/Licenses/License/Text * /DataSets/DataSet/IPRStatements/Disclaimers/Disclaimer/Text * /DataSets/DataSet/IPRStatements/Citations/Citation/Text * /DataSets/DataSet/IPRStatements/Acknowledgements/Acknowledgement/Text Unitdata * /DataSets/DataSet/Units/Unit/DateLastEdited * /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/InformalNameString * /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/InformalNameString[@language] * /DataSets/DataSet/Units/Unit/Identifications/Identification/Identifiers/Identifier/IdentifierPersonName/FullName * /DataSets/DataSet/Units/Unit/Identifications/Identification/Date/ISODateTimeBegin * /DataSets/DataSet/Units/Unit/KindOfUnit * /DataSets/DataSet/Units/Unit/SpecimenUnit/NomenclaturalTypeDesignations/NomenclaturalTypeDesignation/TypifiedName/FullScientificNameString * /DataSets/DataSet/Units/Unit/SpecimenUnit/NomenclaturalTypeDesignations/NomenclaturalTypeDesignation/TypeStatus */DataSets/DataSet/Units/Unit/SpecimenUnit/NomenclaturalTypeDesignations/NomenclaturalTypeDesignation/NomenclaturalReference/TitleCitation * /DataSets/DataSet/Units/Unit/SpecimenUnit/NomenclaturalTypeDesignations/NomenclaturalTypeDesignation/NomenclaturalReference/CitationDetail * /DataSets/DataSet/Units/Unit/MultiMediaObjects/MultiMediaObject/FileURI * /DataSets/DataSet/Units/Unit/MultiMediaObjects/MultiMediaObject/Comment * /DataSets/DataSet/Units/Unit/MultiMediaObjects/MultiMediaObject/Context * /DataSets/DataSet/Units/Unit/Gathering/Project/ProjectTitle * /DataSets/DataSet/Units/Unit/Gathering/NamedAreas/NamedArea/AreaName * /DataSets/DataSet/Units/Unit/Gathering/NamedAreas/NamedArea/AreaClass * /DataSets/DataSet/Units/Unit/Gathering/Altitude/MeasurementOrFactAtomised/LowerValue * /DataSets/DataSet/Units/Unit/Gathering/Altitude/MeasurementOrFactAtomised/UnitOfMeasurement * /DataSets/DataSet/Units/Unit/Gathering/Depth/MeasurementOrFactAtomised/LowerValue * /DataSets/DataSet/Units/Unit/Gathering/Depth/MeasurementOrFactAtomised/UnitOfMeasurement * /DataSets/DataSet/Units/Unit/Gathering/Biotope/Name * /DataSets/DataSet/Units/Unit/CollectorsFieldNumber * /DataSets/DataSet/Units/Unit/Sex * /DataSets/DataSet/Units/Unit/Age * /DataSets/DataSet/Units/Unit/Notes * /DataSets/DataSet/Units/Unit/RecordURI 2dcf8a127cfefad3039cb4652f3c03091796afbd 57 56 2011-06-22T13:53:41Z JoergHoletschek 3 wikitext text/x-wiki This page lists the most commonly used ABCD 2.06 concepts. The ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a full list of ABCD concepts [http://wiki.tdwg.org/twiki/bin/view/ABCD/AbcdConcepts here]. == Mandatory == List of mandatory concepts. Without them being mapped the Provider Software will not be able to create valid ABCD document: Metadata * /DataSets/DataSet/ContentContacts/ContentContact/Name * /DataSets/DataSet/TechnicalContacts/TechnicalContact/Name * /DataSets/DataSet/Metadata/Description/Representation/Title * /DataSets/DataSet/Metadata/Description/Representation[@language] * /DataSets/DataSet/Metadata/RevisionData/DateModified Unitdata * /DataSets/DataSet/Units/Unit/SourceInstitutionID * /DataSets/DataSet/Units/Unit/SourceID * /DataSets/DataSet/Units/Unit/UnitID == Highly Recommended == List of recommended ABCD concepts. If you can provide this data, please do so, they're commonly used entry points to your data. Metadata * /DataSets/DataSet/ContentContacts/ContentContact/Email * /DataSets/DataSet/ContentContacts/ContentContact/Address * /DataSets/DataSet/TechnicalContacts/TechnicalContact/Email * /DataSets/DataSet/TechnicalContacts/TechnicalContact/Address * /DataSets/DataSet/Metadata/Description/Representation/Details * /DataSets/DataSet/Metadata/Owners/Owner/Organisation/Name/Representation/Text * /DataSets/DataSet/Metadata/Owners/Owner/Person/FullName * /DataSets/DataSet/Metadata/Owners/Owner/URIs/URL * /DataSets/DataSet/Metadata/Owners/Owner/LogoURI Unitdata * /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString * /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/HigherTaxa/HigherTaxon/HigherTaxonName * /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/HigherTaxa/HigherTaxon/HigherTaxonRank * /DataSets/DataSet/Units/Unit/Identifications/Identification/PreferredFlag * /DataSets/DataSet/Units/Unit/RecordBasis * /DataSets/DataSet/Units/Unit/Gathering/DateTime/ISODateTimeBegin * /DataSets/DataSet/Units/Unit/Gathering/Agents/GatheringAgent/Person/FullName * /DataSets/DataSet/Units/Unit/Gathering/LocalityText * /DataSets/DataSet/Units/Unit/Gathering/Country/ISO3166Code * /DataSets/DataSet/Units/Unit/Gathering/Country/Name * /DataSets/DataSet/Units/Unit/Gathering/SiteCoordinateSets/SiteCoordinates/CoordinatesLatLong/LongitudeDecimal */DataSets/DataSet/Units/Unit/Gathering/SiteCoordinateSets/SiteCoordinates/CoordinatesLatLong/LatitudeDecimal == Commonly Used == List of further commonly mapped ABCD concepts. Metadata * /DataSets/DataSet/IPRStatements/TermsOfUseStatements/TermsOfUse/Text * /DataSets/DataSet/IPRStatements/Copyrights/Copyright/Text * /DataSets/DataSet/IPRStatements/IPRDeclarations/IPRDeclaration/Text * /DataSets/DataSet/IPRStatements/Licenses/License/Text * /DataSets/DataSet/IPRStatements/Disclaimers/Disclaimer/Text * /DataSets/DataSet/IPRStatements/Citations/Citation/Text * /DataSets/DataSet/IPRStatements/Acknowledgements/Acknowledgement/Text Unitdata * /DataSets/DataSet/Units/Unit/DateLastEdited * /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/InformalNameString * /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/InformalNameString[@language] * /DataSets/DataSet/Units/Unit/Identifications/Identification/Identifiers/Identifier/IdentifierPersonName/FullName * /DataSets/DataSet/Units/Unit/Identifications/Identification/Date/ISODateTimeBegin * /DataSets/DataSet/Units/Unit/KindOfUnit * /DataSets/DataSet/Units/Unit/SpecimenUnit/NomenclaturalTypeDesignations/NomenclaturalTypeDesignation/TypifiedName/FullScientificNameString * /DataSets/DataSet/Units/Unit/SpecimenUnit/NomenclaturalTypeDesignations/NomenclaturalTypeDesignation/TypeStatus */DataSets/DataSet/Units/Unit/SpecimenUnit/NomenclaturalTypeDesignations/NomenclaturalTypeDesignation/NomenclaturalReference/TitleCitation * /DataSets/DataSet/Units/Unit/SpecimenUnit/NomenclaturalTypeDesignations/NomenclaturalTypeDesignation/NomenclaturalReference/CitationDetail * /DataSets/DataSet/Units/Unit/MultiMediaObjects/MultiMediaObject/FileURI * /DataSets/DataSet/Units/Unit/MultiMediaObjects/MultiMediaObject/Comment * /DataSets/DataSet/Units/Unit/MultiMediaObjects/MultiMediaObject/Context * /DataSets/DataSet/Units/Unit/Gathering/Project/ProjectTitle * /DataSets/DataSet/Units/Unit/Gathering/NamedAreas/NamedArea/AreaName * /DataSets/DataSet/Units/Unit/Gathering/NamedAreas/NamedArea/AreaClass * /DataSets/DataSet/Units/Unit/Gathering/Altitude/MeasurementOrFactAtomised/LowerValue * /DataSets/DataSet/Units/Unit/Gathering/Altitude/MeasurementOrFactAtomised/UnitOfMeasurement * /DataSets/DataSet/Units/Unit/Gathering/Depth/MeasurementOrFactAtomised/LowerValue * /DataSets/DataSet/Units/Unit/Gathering/Depth/MeasurementOrFactAtomised/UnitOfMeasurement * /DataSets/DataSet/Units/Unit/Gathering/Biotope/Name * /DataSets/DataSet/Units/Unit/CollectorsFieldNumber * /DataSets/DataSet/Units/Unit/Sex * /DataSets/DataSet/Units/Unit/Age * /DataSets/DataSet/Units/Unit/Notes * /DataSets/DataSet/Units/Unit/RecordURI c0fe9da0bc65e0a9f715338047587c98d5150e9f 0 23 58 2011-06-22T14:03:29Z JoergHoletschek 3 Created page with "<h2>Version 2.6</h2> <h3>Version 2.6.1 [upcoming]</h3> * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. ===Versio..." wikitext text/x-wiki <h2>Version 2.6</h2> <h3>Version 2.6.1 [upcoming]</h3> * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14, withdrawn] === === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. See example [http://ww3.bgbm.org/bps2/ScreenShots?action=AttachFile&do=get&target=config_tool4.gif here] * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: * Internet Explorer compatability fixed (no tablealias names were saved) * CMF filter display fixed. Colored green now * Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly * Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included * Bug fixing * Configtool * Lot of little bugs solved. * Make possible to map attributes * Be able to delete mappings * Alert user when several CMFs with the same namespace exist for a datasource * Allow full xpaths to be entered directly into the add path form * allow datasource admin to delete all serialized objects * !QueryTool * More inteligent XSLT for processing * !PyWrapper * Solved error with wrapper not presenting attributes * Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support = Version 1 history = === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: . Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. . Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton . for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml . and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 57f9d34ffc0268040a0ecff7baed2fee1cdbeccb 59 58 2011-06-22T14:04:57Z JoergHoletschek 3 wikitext text/x-wiki <h2>Version 2.6<h2> <h3>Version 2.6.1 [upcoming]<h3> * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14, withdrawn] === === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. See example [http://ww3.bgbm.org/bps2/ScreenShots?action=AttachFile&do=get&target=config_tool4.gif here] * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: * Internet Explorer compatability fixed (no tablealias names were saved) * CMF filter display fixed. Colored green now * Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly * Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included * Bug fixing * Configtool * Lot of little bugs solved. * Make possible to map attributes * Be able to delete mappings * Alert user when several CMFs with the same namespace exist for a datasource * Allow full xpaths to be entered directly into the add path form * allow datasource admin to delete all serialized objects * !QueryTool * More inteligent XSLT for processing * !PyWrapper * Solved error with wrapper not presenting attributes * Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support = Version 1 history = === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: . Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. . Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton . for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml . and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool b04a256f95e9cf638d5577ceee2b3eb74c9b09fb 60 59 2011-06-22T14:24:10Z JoergHoletschek 3 wikitext text/x-wiki == Version 2 History == ===Version 2.6.1 [upcoming]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14, withdrawn] === === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. See example [http://ww3.bgbm.org/bps2/ScreenShots?action=AttachFile&do=get&target=config_tool4.gif here] * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 6923382a5d52f26c6def32a59c4d71bc755b65c8 0 24 61 2011-06-23T14:12:19Z JoergHoletschek 3 Created page with "This tutorial explains in detail how to set up a datasource. A BioCASe Data Source Publishing a database with BioCASe means it will be made accessible through a BioCASe web serv..." wikitext text/x-wiki This tutorial explains in detail how to set up a datasource. A BioCASe Data Source Publishing a database with BioCASe means it will be made accessible through a BioCASe web service, which will be used later by the biodiversity network(s) you want to join to access to your data. For example, the GBIF Harvester will send a BioCASe Scan request to that web service to get a list of all scientific names kept in the database, and subsequently several BioCASe Search requests to load all records for these names. For each database you want to publish, you need to set up a BioCASe data source (do not mix that up with ODBC data sources on Windows machines). The resulting BioCASe web service is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source. So if you made your installation available at http://www.foobar.org/biocase during the installation process and set up a data source named Herbar, the URL of the BioCASe web service would be http://www.foobar.org/biocase/pywrapper.cgi?dsa=Herbar (data source names can be case sensitive, depending on your server’s operating system). Creating a New Data Source The start page of the BPS will always list the data sources currently configured, that means the web services offered by that BioCASe installation. On a vanilla installation, it will be empty: Creating a new data source can be done in the configuration tool. Choose Config Tool on the start page, then System Administration. Log in with the password you set during the installation process (if you didn’t change that yet, the default is ACDC). In the configuration screen, the sections Data Sources and Recycle Bin can be used to create, remove und un-remove a data source: For now, we want to create a new, empty data source. Enter a name – without any spaces – in the text box and click Create DSA (remember that the datasource name is case sensitive on Linux machines). After a second, it will appear in the list of existing data sources with a clickable link. By doing this, you created the first BioCASe web service for your installation. Clicking “Remove” will uninstall this service again (putting it in the recycle bin, so you can restore it later). Tip: If you select abcdmetadata in the Template drop-down list, your data source will be created with an ABCD mapping for all the metadata elements stored in the metadata table mentioned in the ___Database preparation___ guide. So if you did use the templates in sampledb/abcdmetadata to create the metadata table in your database, you can use this template to spare the hassle of mapping this table and start right away with your occurrence data. Setting up the database connection Clicking on the data source you’ve just created will take you to the datasource configuration tool. Note that the Schemas section will be empty if you’ve created an empty data source or list two schemas (ABCD 1.2 and 2.06) if you created the datasource based on the abcdmetadata template: The link on the Database Connection section will take you to the connection parameters dialog. Fill in the values for your database, press Save and Hope for the connection to turn into a nice green OK: DBMS: Select the entry appropriate for your collection. The BPS can directly connect to database servers (MySQL, Postgres, DB2, SQL Server, and Oracle) and to Desktop-DBMS like Access and FoxPro through ODBC. ODBC is only available on Windows, so if you plan to use it, BioCASe must be installed on a Windows Server. Host: Enter the IP address of the database server. If you’re connection through ODBC, you must leave this empty. Instead, use Windows’ ODBC Administrator to set up a System (!) Datasource; the server or file name of the DBMS will be stored in the ODBC configuration. Database: Enter the name of the ODBC datasource if you’re using ODBC, the name of the database if you’re directly connecting to a server. User/Password: Database credentials to be used. As written in the ___DB Preparation Guide___, you should create a user with just Read access for this. Encoding: The encoding of your database. If you’re not sure, leave the default, you can change it if you run into problems later. If you know your database is using a different encoding than latin-1 (which is typically the default for western languages), maybe UTF-8, select the appropriate entry in the drop-down list. Once you’ve entered the correct values, press “Save”. Click “Overview” in the link list on the top of the page to return to the datasource configuration page. If the connection status doesn’t turn into OK (and doesn’t after you’ve returned to the Overview page), please read the __Debug guide__ about how to trace the problem. Setting up the Database Structure Clicking on “Edit DB Structure” on the Datasource Overview page will show the DB structure setup. Make sure the connection status at the top of the page still is OK. In a first step, you will add all tables/views that hold information you want to publish with BioCASe. You can ignore all database tables storing only data you don’t plan to map to ABCD. Same holds true if you use cache tables (as explained in the ___DB preparation guide___): Only the cache tables need to be set up on the database structure page. In a second step, you will specify the foreign key references between these tables. First step: Adding all tables/views with their primary keys To add a table or view, just select it in the drop-down list. The name of the table will be filled in the Alias text box (you will use this alias later in the mapping process), feel free to change it into something that describes the contents of that tables (for example specimen for the root specimen table). Press Add to insert the table into the list. Before continuing to the next table, select the primary key in the list of attributes under Primary Key Atrribute(s). In case the data type is a string and not an integer, select text from the drop-down list. Press Add to add the primary key. If the primary key consists of several columns, you can add several primary key attributes. Redo this step (adding the tables first, then the primary key(s)) for all tables you want to publish data from. Always remember to set the correct primary key column(s), even if they’re not defined as primary keys in the database. Tables without a primary key cannot be used by the Provider Software and will be removed upon saving! Second step: Specifying the foreign keys Once you’ve added all tables, you can define the relations between these tables by defining foreign keys. In order to do this, go to the Foreign Key(s) column of the table that holds a foreign key reference to another table, select the alias of the referenced table and press Add. Select the attribute that stores the foreign key and change the data type if it is not integer, then press Add. You must specify as many attributes for the foreign keys as the primary key of the referenced table has, otherwise the Provider Software won’t be able to create a valid join between the tables. Also, they must be in the same order and have identical data types. Redo this for all tables until you’ve specified all foreign key references. If you’ve installed the Graphviz package, the Provider Software will create a graph visualizing the current database structure. Each table will be visible as a node with the primary key(s) listed. References between tables will be represented by arrows, the foreign key(s) listed in the referring table: Saving, Editing an existing Setup You can change aliases of tables you’ve already added to the list. To do so, go to the alias text box of that table and type in a different name. Upon exiting the text box, all references to this alias will be updated in the table setup; also the graph will reflect the changes. But caution: Schema mappings use these aliases to bind schema elements to tables and columns, if you change an alias you’ve already used in a schema mapping, you will break this binding! So it is advisable to decide for meaningful alias names and keep them stable once you’ve started the mapping process. Once you’ve added a table/view to the table list, you can’t change the underlying table, since this would invalidate all the primary and foreign key attributes. If you’ve added the wrong table, remove it by using the trash can symbol next to “Delete Alias” and add the correct table/view. If you’ve finished the table setup and the graph displays the model of your database correctly, save the configuration by Pressing Save at the bottom of the page. This is important; navigating away from the page without saving will cause your changes to be lost (even though you will be asked for confirmation)! It is advisable to save the table setup every now and then, for example once you’ve added all tables and primary keys. If you edit an existing table setup and happen to realize you’ve messed it up, you can revert to the state when you’ve pressed Save for the last time by using the Revert button at the bottom. But be careful: All unsaved changes will be lost. Note: The retrieval of tables/view/column names is currently only supported for the following DBMS: Microsoft SQL Server, MySQL, Access, Postgres and FoxPro. If you use a different DBMS, you will see a simple text box instead of a drop-down-list, where you’ll have to type in the name of the table or column you want to add. fe7edc89514ad47d5f473e0b9aa05cedc2ab7256 62 61 2011-06-23T14:14:37Z JoergHoletschek 3 wikitext text/x-wiki This tutorial explains in detail how to set up a datasource. == A BioCASe Data Source == Publishing a database with BioCASe means it will be made accessible through a BioCASe web service, which will be used later by the biodiversity network(s) you want to join to access to your data. For example, the GBIF Harvester will send a BioCASe Scan request to that web service to get a list of all scientific names kept in the database, and subsequently several BioCASe Search requests to load all records for these names. For each database you want to publish, you need to set up a BioCASe data source (do not mix that up with ODBC data sources on Windows machines). The resulting BioCASe web service is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source. So if you made your installation available at http://www.foobar.org/biocase during the installation process and set up a data source named Herbar, the URL of the BioCASe web service would be http://www.foobar.org/biocase/pywrapper.cgi?dsa=Herbar (data source names can be case sensitive, depending on your server’s operating system). == Creating a New Data Source == The start page of the BPS will always list the data sources currently configured, that means the web services offered by that BioCASe installation. On a vanilla installation, it will be empty: Creating a new data source can be done in the configuration tool. Choose Config Tool on the start page, then System Administration. Log in with the password you set during the installation process (if you didn’t change that yet, the default is ACDC). In the configuration screen, the sections Data Sources and Recycle Bin can be used to create, remove und un-remove a data source: For now, we want to create a new, empty data source. Enter a name – without any spaces – in the text box and click Create DSA (remember that the datasource name is case sensitive on Linux machines). After a second, it will appear in the list of existing data sources with a clickable link. By doing this, you created the first BioCASe web service for your installation. Clicking “Remove” will uninstall this service again (putting it in the recycle bin, so you can restore it later). Tip: If you select abcdmetadata in the Template drop-down list, your data source will be created with an ABCD mapping for all the metadata elements stored in the metadata table mentioned in the ___Database preparation___ guide. So if you did use the templates in sampledb/abcdmetadata to create the metadata table in your database, you can use this template to spare the hassle of mapping this table and start right away with your occurrence data. == Setting up the database connection == Clicking on the data source you’ve just created will take you to the datasource configuration tool. Note that the Schemas section will be empty if you’ve created an empty data source or list two schemas (ABCD 1.2 and 2.06) if you created the datasource based on the abcdmetadata template: The link on the Database Connection section will take you to the connection parameters dialog. Fill in the values for your database, press Save and Hope for the connection to turn into a nice green OK: DBMS: Select the entry appropriate for your collection. The BPS can directly connect to database servers (MySQL, Postgres, DB2, SQL Server, and Oracle) and to Desktop-DBMS like Access and FoxPro through ODBC. ODBC is only available on Windows, so if you plan to use it, BioCASe must be installed on a Windows Server. Host: Enter the IP address of the database server. If you’re connection through ODBC, you must leave this empty. Instead, use Windows’ ODBC Administrator to set up a System (!) Datasource; the server or file name of the DBMS will be stored in the ODBC configuration. Database: Enter the name of the ODBC datasource if you’re using ODBC, the name of the database if you’re directly connecting to a server. User/Password: Database credentials to be used. As written in the ___DB Preparation Guide___, you should create a user with just Read access for this. Encoding: The encoding of your database. If you’re not sure, leave the default, you can change it if you run into problems later. If you know your database is using a different encoding than latin-1 (which is typically the default for western languages), maybe UTF-8, select the appropriate entry in the drop-down list. Once you’ve entered the correct values, press “Save”. Click “Overview” in the link list on the top of the page to return to the datasource configuration page. If the connection status doesn’t turn into OK (and doesn’t after you’ve returned to the Overview page), please read the __Debug guide__ about how to trace the problem. == Setting up the Database Structure == Clicking on “Edit DB Structure” on the Datasource Overview page will show the DB structure setup. Make sure the connection status at the top of the page still is OK. In a first step, you will add all tables/views that hold information you want to publish with BioCASe. You can ignore all database tables storing only data you don’t plan to map to ABCD. Same holds true if you use cache tables (as explained in the ___DB preparation guide___): Only the cache tables need to be set up on the database structure page. In a second step, you will specify the foreign key references between these tables. === Adding Tables/Views with their Primary Keys To add a table or view, just select it in the drop-down list. The name of the table will be filled in the Alias text box (you will use this alias later in the mapping process), feel free to change it into something that describes the contents of that tables (for example specimen for the root specimen table). Press Add to insert the table into the list. Before continuing to the next table, select the primary key in the list of attributes under Primary Key Atrribute(s). In case the data type is a string and not an integer, select text from the drop-down list. Press Add to add the primary key. If the primary key consists of several columns, you can add several primary key attributes. Redo this step (adding the tables first, then the primary key(s)) for all tables you want to publish data from. Always remember to set the correct primary key column(s), even if they’re not defined as primary keys in the database. Tables without a primary key cannot be used by the Provider Software and will be removed upon saving! === Specifying Foreign Key References === Once you’ve added all tables, you can define the relations between these tables by defining foreign keys. In order to do this, go to the Foreign Key(s) column of the table that holds a foreign key reference to another table, select the alias of the referenced table and press Add. Select the attribute that stores the foreign key and change the data type if it is not integer, then press Add. You must specify as many attributes for the foreign keys as the primary key of the referenced table has, otherwise the Provider Software won’t be able to create a valid join between the tables. Also, they must be in the same order and have identical data types. Redo this for all tables until you’ve specified all foreign key references. If you’ve installed the Graphviz package, the Provider Software will create a graph visualizing the current database structure. Each table will be visible as a node with the primary key(s) listed. References between tables will be represented by arrows, the foreign key(s) listed in the referring table: === Saving, Editing an Existing Setup === You can change aliases of tables you’ve already added to the list. To do so, go to the alias text box of that table and type in a different name. Upon exiting the text box, all references to this alias will be updated in the table setup; also the graph will reflect the changes. But caution: Schema mappings use these aliases to bind schema elements to tables and columns, if you change an alias you’ve already used in a schema mapping, you will break this binding! So it is advisable to decide for meaningful alias names and keep them stable once you’ve started the mapping process. Once you’ve added a table/view to the table list, you can’t change the underlying table, since this would invalidate all the primary and foreign key attributes. If you’ve added the wrong table, remove it by using the trash can symbol next to “Delete Alias” and add the correct table/view. If you’ve finished the table setup and the graph displays the model of your database correctly, save the configuration by Pressing Save at the bottom of the page. This is important; navigating away from the page without saving will cause your changes to be lost (even though you will be asked for confirmation)! It is advisable to save the table setup every now and then, for example once you’ve added all tables and primary keys. If you edit an existing table setup and happen to realize you’ve messed it up, you can revert to the state when you’ve pressed Save for the last time by using the Revert button at the bottom. But be careful: All unsaved changes will be lost. Note: The retrieval of tables/view/column names is currently only supported for the following DBMS: Microsoft SQL Server, MySQL, Access, Postgres and FoxPro. If you use a different DBMS, you will see a simple text box instead of a drop-down-list, where you’ll have to type in the name of the table or column you want to add. 545677a745f6bdc7eddea5cecb4dd7d3c0a66a95 63 62 2011-06-23T14:20:36Z JoergHoletschek 3 /* Creating a New Data Source */ wikitext text/x-wiki This tutorial explains in detail how to set up a datasource. == A BioCASe Data Source == Publishing a database with BioCASe means it will be made accessible through a BioCASe web service, which will be used later by the biodiversity network(s) you want to join to access to your data. For example, the GBIF Harvester will send a BioCASe Scan request to that web service to get a list of all scientific names kept in the database, and subsequently several BioCASe Search requests to load all records for these names. For each database you want to publish, you need to set up a BioCASe data source (do not mix that up with ODBC data sources on Windows machines). The resulting BioCASe web service is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source. So if you made your installation available at http://www.foobar.org/biocase during the installation process and set up a data source named Herbar, the URL of the BioCASe web service would be http://www.foobar.org/biocase/pywrapper.cgi?dsa=Herbar (data source names can be case sensitive, depending on your server’s operating system). == Creating a New Data Source == The start page of the BPS will always list the data sources currently configured, that means the web services offered by that BioCASe installation. On a vanilla installation, it will be empty: [[File:instBiocaseStartpage.png|500px]] Creating a new data source can be done in the configuration tool. Choose <small>Config Tool</small> on the start page, then <small>System Administration</small>. Log in with the password you set during the installation process (if you didn’t change that yet, the default is ''ACDC''). In the configuration screen, the sections <small>Data Sources</small> and <small>Recycle Bin</small> can be used to create, remove und un-remove a data source: [[File:dsaCreateDatasourceEmpty.png]] For now, we want to create a new, empty data source. Enter a name – without any spaces – in the text box and click <small>Create DSA</small> (remember that the datasource name is case sensitive on Linux machines). After a second, it will appear in the list of existing data sources with a clickable link. By doing this, you created the first BioCASe web service for your installation. Clicking <small>Remove</small> will uninstall this service again (putting it in the recycle bin, so you can restore it later). Tip: If you select <small>abcdmetadata</small> in the template drop-down list, your data source will be created with an ABCD mapping for all the metadata elements stored in the metadata table mentioned in the ___Database preparation___ guide. So if you did use the templates in ''sampledb/abcdmetadata'' to create the metadata table in your database, you can use this template to spare the hassle of mapping this table and start right away with your occurrence data. == Setting up the database connection == Clicking on the data source you’ve just created will take you to the datasource configuration tool. Note that the Schemas section will be empty if you’ve created an empty data source or list two schemas (ABCD 1.2 and 2.06) if you created the datasource based on the abcdmetadata template: The link on the Database Connection section will take you to the connection parameters dialog. Fill in the values for your database, press Save and Hope for the connection to turn into a nice green OK: DBMS: Select the entry appropriate for your collection. The BPS can directly connect to database servers (MySQL, Postgres, DB2, SQL Server, and Oracle) and to Desktop-DBMS like Access and FoxPro through ODBC. ODBC is only available on Windows, so if you plan to use it, BioCASe must be installed on a Windows Server. Host: Enter the IP address of the database server. If you’re connection through ODBC, you must leave this empty. Instead, use Windows’ ODBC Administrator to set up a System (!) Datasource; the server or file name of the DBMS will be stored in the ODBC configuration. Database: Enter the name of the ODBC datasource if you’re using ODBC, the name of the database if you’re directly connecting to a server. User/Password: Database credentials to be used. As written in the ___DB Preparation Guide___, you should create a user with just Read access for this. Encoding: The encoding of your database. If you’re not sure, leave the default, you can change it if you run into problems later. If you know your database is using a different encoding than latin-1 (which is typically the default for western languages), maybe UTF-8, select the appropriate entry in the drop-down list. Once you’ve entered the correct values, press “Save”. Click “Overview” in the link list on the top of the page to return to the datasource configuration page. If the connection status doesn’t turn into OK (and doesn’t after you’ve returned to the Overview page), please read the __Debug guide__ about how to trace the problem. == Setting up the Database Structure == Clicking on “Edit DB Structure” on the Datasource Overview page will show the DB structure setup. Make sure the connection status at the top of the page still is OK. In a first step, you will add all tables/views that hold information you want to publish with BioCASe. You can ignore all database tables storing only data you don’t plan to map to ABCD. Same holds true if you use cache tables (as explained in the ___DB preparation guide___): Only the cache tables need to be set up on the database structure page. In a second step, you will specify the foreign key references between these tables. === Adding Tables/Views with their Primary Keys To add a table or view, just select it in the drop-down list. The name of the table will be filled in the Alias text box (you will use this alias later in the mapping process), feel free to change it into something that describes the contents of that tables (for example specimen for the root specimen table). Press Add to insert the table into the list. Before continuing to the next table, select the primary key in the list of attributes under Primary Key Atrribute(s). In case the data type is a string and not an integer, select text from the drop-down list. Press Add to add the primary key. If the primary key consists of several columns, you can add several primary key attributes. Redo this step (adding the tables first, then the primary key(s)) for all tables you want to publish data from. Always remember to set the correct primary key column(s), even if they’re not defined as primary keys in the database. Tables without a primary key cannot be used by the Provider Software and will be removed upon saving! === Specifying Foreign Key References === Once you’ve added all tables, you can define the relations between these tables by defining foreign keys. In order to do this, go to the Foreign Key(s) column of the table that holds a foreign key reference to another table, select the alias of the referenced table and press Add. Select the attribute that stores the foreign key and change the data type if it is not integer, then press Add. You must specify as many attributes for the foreign keys as the primary key of the referenced table has, otherwise the Provider Software won’t be able to create a valid join between the tables. Also, they must be in the same order and have identical data types. Redo this for all tables until you’ve specified all foreign key references. If you’ve installed the Graphviz package, the Provider Software will create a graph visualizing the current database structure. Each table will be visible as a node with the primary key(s) listed. References between tables will be represented by arrows, the foreign key(s) listed in the referring table: === Saving, Editing an Existing Setup === You can change aliases of tables you’ve already added to the list. To do so, go to the alias text box of that table and type in a different name. Upon exiting the text box, all references to this alias will be updated in the table setup; also the graph will reflect the changes. But caution: Schema mappings use these aliases to bind schema elements to tables and columns, if you change an alias you’ve already used in a schema mapping, you will break this binding! So it is advisable to decide for meaningful alias names and keep them stable once you’ve started the mapping process. Once you’ve added a table/view to the table list, you can’t change the underlying table, since this would invalidate all the primary and foreign key attributes. If you’ve added the wrong table, remove it by using the trash can symbol next to “Delete Alias” and add the correct table/view. If you’ve finished the table setup and the graph displays the model of your database correctly, save the configuration by Pressing Save at the bottom of the page. This is important; navigating away from the page without saving will cause your changes to be lost (even though you will be asked for confirmation)! It is advisable to save the table setup every now and then, for example once you’ve added all tables and primary keys. If you edit an existing table setup and happen to realize you’ve messed it up, you can revert to the state when you’ve pressed Save for the last time by using the Revert button at the bottom. But be careful: All unsaved changes will be lost. Note: The retrieval of tables/view/column names is currently only supported for the following DBMS: Microsoft SQL Server, MySQL, Access, Postgres and FoxPro. If you use a different DBMS, you will see a simple text box instead of a drop-down-list, where you’ll have to type in the name of the table or column you want to add. 00ea3d32a2377ea28da913a9cf51a266ca687a03 65 63 2011-06-23T14:25:38Z JoergHoletschek 3 /* Creating a New Data Source */ wikitext text/x-wiki This tutorial explains in detail how to set up a datasource. == A BioCASe Data Source == Publishing a database with BioCASe means it will be made accessible through a BioCASe web service, which will be used later by the biodiversity network(s) you want to join to access to your data. For example, the GBIF Harvester will send a BioCASe Scan request to that web service to get a list of all scientific names kept in the database, and subsequently several BioCASe Search requests to load all records for these names. For each database you want to publish, you need to set up a BioCASe data source (do not mix that up with ODBC data sources on Windows machines). The resulting BioCASe web service is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source. So if you made your installation available at http://www.foobar.org/biocase during the installation process and set up a data source named Herbar, the URL of the BioCASe web service would be http://www.foobar.org/biocase/pywrapper.cgi?dsa=Herbar (data source names can be case sensitive, depending on your server’s operating system). == Creating a New Data Source == The start page of the BPS will always list the data sources currently configured, that means the web services offered by that BioCASe installation. On a vanilla installation, it will be empty: [[File:instBiocaseStartpage.png|500px]] Creating a new data source can be done in the configuration tool. Choose <small>Config Tool</small> on the start page, then <small>System Administration</small>. Log in with the password you set during the installation process (if you didn’t change that yet, the default is ''ACDC''). In the configuration screen, the sections <small>Data Sources</small> and <small>Recycle Bin</small> can be used to create, remove und un-remove a data source: [[File:dsaCreateDatasourceEmpty.png|500px]] For now, we want to create a new, empty data source. Enter a name – without any spaces – in the text box and click <small>Create DSA</small> (remember that the datasource name is case sensitive on Linux machines). After a second, it will appear in the list of existing data sources with a clickable link. By doing this, you created the first BioCASe web service for your installation. Clicking <small>Remove</small> will uninstall this service again (putting it in the recycle bin, so you can restore it later). Tip: If you select <small>abcdmetadata</small> in the template drop-down list, your data source will be created with an ABCD mapping for all the metadata elements stored in the metadata table mentioned in the ___Database preparation___ guide. So if you did use the templates in ''sampledb/abcdmetadata'' to create the metadata table in your database, you can use this template to spare the hassle of mapping this table and start right away with your occurrence data. == Setting up the database connection == Clicking on the data source you’ve just created will take you to the datasource configuration tool. Note that the Schemas section will be empty if you’ve created an empty data source or list two schemas (ABCD 1.2 and 2.06) if you created the datasource based on the abcdmetadata template: The link on the Database Connection section will take you to the connection parameters dialog. Fill in the values for your database, press Save and Hope for the connection to turn into a nice green OK: DBMS: Select the entry appropriate for your collection. The BPS can directly connect to database servers (MySQL, Postgres, DB2, SQL Server, and Oracle) and to Desktop-DBMS like Access and FoxPro through ODBC. ODBC is only available on Windows, so if you plan to use it, BioCASe must be installed on a Windows Server. Host: Enter the IP address of the database server. If you’re connection through ODBC, you must leave this empty. Instead, use Windows’ ODBC Administrator to set up a System (!) Datasource; the server or file name of the DBMS will be stored in the ODBC configuration. Database: Enter the name of the ODBC datasource if you’re using ODBC, the name of the database if you’re directly connecting to a server. User/Password: Database credentials to be used. As written in the ___DB Preparation Guide___, you should create a user with just Read access for this. Encoding: The encoding of your database. If you’re not sure, leave the default, you can change it if you run into problems later. If you know your database is using a different encoding than latin-1 (which is typically the default for western languages), maybe UTF-8, select the appropriate entry in the drop-down list. Once you’ve entered the correct values, press “Save”. Click “Overview” in the link list on the top of the page to return to the datasource configuration page. If the connection status doesn’t turn into OK (and doesn’t after you’ve returned to the Overview page), please read the __Debug guide__ about how to trace the problem. == Setting up the Database Structure == Clicking on “Edit DB Structure” on the Datasource Overview page will show the DB structure setup. Make sure the connection status at the top of the page still is OK. In a first step, you will add all tables/views that hold information you want to publish with BioCASe. You can ignore all database tables storing only data you don’t plan to map to ABCD. Same holds true if you use cache tables (as explained in the ___DB preparation guide___): Only the cache tables need to be set up on the database structure page. In a second step, you will specify the foreign key references between these tables. === Adding Tables/Views with their Primary Keys To add a table or view, just select it in the drop-down list. The name of the table will be filled in the Alias text box (you will use this alias later in the mapping process), feel free to change it into something that describes the contents of that tables (for example specimen for the root specimen table). Press Add to insert the table into the list. Before continuing to the next table, select the primary key in the list of attributes under Primary Key Atrribute(s). In case the data type is a string and not an integer, select text from the drop-down list. Press Add to add the primary key. If the primary key consists of several columns, you can add several primary key attributes. Redo this step (adding the tables first, then the primary key(s)) for all tables you want to publish data from. Always remember to set the correct primary key column(s), even if they’re not defined as primary keys in the database. Tables without a primary key cannot be used by the Provider Software and will be removed upon saving! === Specifying Foreign Key References === Once you’ve added all tables, you can define the relations between these tables by defining foreign keys. In order to do this, go to the Foreign Key(s) column of the table that holds a foreign key reference to another table, select the alias of the referenced table and press Add. Select the attribute that stores the foreign key and change the data type if it is not integer, then press Add. You must specify as many attributes for the foreign keys as the primary key of the referenced table has, otherwise the Provider Software won’t be able to create a valid join between the tables. Also, they must be in the same order and have identical data types. Redo this for all tables until you’ve specified all foreign key references. If you’ve installed the Graphviz package, the Provider Software will create a graph visualizing the current database structure. Each table will be visible as a node with the primary key(s) listed. References between tables will be represented by arrows, the foreign key(s) listed in the referring table: === Saving, Editing an Existing Setup === You can change aliases of tables you’ve already added to the list. To do so, go to the alias text box of that table and type in a different name. Upon exiting the text box, all references to this alias will be updated in the table setup; also the graph will reflect the changes. But caution: Schema mappings use these aliases to bind schema elements to tables and columns, if you change an alias you’ve already used in a schema mapping, you will break this binding! So it is advisable to decide for meaningful alias names and keep them stable once you’ve started the mapping process. Once you’ve added a table/view to the table list, you can’t change the underlying table, since this would invalidate all the primary and foreign key attributes. If you’ve added the wrong table, remove it by using the trash can symbol next to “Delete Alias” and add the correct table/view. If you’ve finished the table setup and the graph displays the model of your database correctly, save the configuration by Pressing Save at the bottom of the page. This is important; navigating away from the page without saving will cause your changes to be lost (even though you will be asked for confirmation)! It is advisable to save the table setup every now and then, for example once you’ve added all tables and primary keys. If you edit an existing table setup and happen to realize you’ve messed it up, you can revert to the state when you’ve pressed Save for the last time by using the Revert button at the bottom. But be careful: All unsaved changes will be lost. Note: The retrieval of tables/view/column names is currently only supported for the following DBMS: Microsoft SQL Server, MySQL, Access, Postgres and FoxPro. If you use a different DBMS, you will see a simple text box instead of a drop-down-list, where you’ll have to type in the name of the table or column you want to add. 707c45996b3dfb07d82be58517e303b27d3cb74d 66 65 2011-06-23T14:32:43Z JoergHoletschek 3 /* Setting up the database connection */ wikitext text/x-wiki This tutorial explains in detail how to set up a datasource. == A BioCASe Data Source == Publishing a database with BioCASe means it will be made accessible through a BioCASe web service, which will be used later by the biodiversity network(s) you want to join to access to your data. For example, the GBIF Harvester will send a BioCASe Scan request to that web service to get a list of all scientific names kept in the database, and subsequently several BioCASe Search requests to load all records for these names. For each database you want to publish, you need to set up a BioCASe data source (do not mix that up with ODBC data sources on Windows machines). The resulting BioCASe web service is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source. So if you made your installation available at http://www.foobar.org/biocase during the installation process and set up a data source named Herbar, the URL of the BioCASe web service would be http://www.foobar.org/biocase/pywrapper.cgi?dsa=Herbar (data source names can be case sensitive, depending on your server’s operating system). == Creating a New Data Source == The start page of the BPS will always list the data sources currently configured, that means the web services offered by that BioCASe installation. On a vanilla installation, it will be empty: [[File:instBiocaseStartpage.png|500px]] Creating a new data source can be done in the configuration tool. Choose <small>Config Tool</small> on the start page, then <small>System Administration</small>. Log in with the password you set during the installation process (if you didn’t change that yet, the default is ''ACDC''). In the configuration screen, the sections <small>Data Sources</small> and <small>Recycle Bin</small> can be used to create, remove und un-remove a data source: [[File:dsaCreateDatasourceEmpty.png|500px]] For now, we want to create a new, empty data source. Enter a name – without any spaces – in the text box and click <small>Create DSA</small> (remember that the datasource name is case sensitive on Linux machines). After a second, it will appear in the list of existing data sources with a clickable link. By doing this, you created the first BioCASe web service for your installation. Clicking <small>Remove</small> will uninstall this service again (putting it in the recycle bin, so you can restore it later). Tip: If you select <small>abcdmetadata</small> in the template drop-down list, your data source will be created with an ABCD mapping for all the metadata elements stored in the metadata table mentioned in the ___Database preparation___ guide. So if you did use the templates in ''sampledb/abcdmetadata'' to create the metadata table in your database, you can use this template to spare the hassle of mapping this table and start right away with your occurrence data. == Setting up the database connection == Clicking on the data source you’ve just created will take you to the datasource configuration tool. Note that the <small>Schemas</small> section will be empty if you’ve created an empty data source or list two schemas (ABCD 1.2 and 2.06) if you created the datasource based on the <small>abcdmetadata</small> template: [[File:dsaOverview.png]] The link in the <small>Database Connection</small> section will take you to the connection parameters dialog. Fill in the values for your database, press <small>Save</small> and hope for the connection to turn into a nice green OK: [[File:dsaDBConnection.png]] '''DBMS:''' Select the entry appropriate for your collection. The BPS can directly connect to database servers (MySQL, Postgres, DB2, SQL Server, and Oracle) and to Desktop-DBMS like Access and FoxPro through ODBC. ODBC is only available on Windows, so if you plan to use it, BioCASe must be installed on a Windows Server. '''Host:''' Enter the IP address of the database server. If you’re connection through ODBC, you must leave this empty. Instead, use Windows’ ODBC Administrator to set up a '''System datasource''' (!); the server or file name of the database will be stored in the ODBC configuration. '''Database:''' Enter the name of the ODBC datasource if you’re using ODBC, the name of the database if you’re directly connecting to a server. '''User/Password:''' Database credentials to be used. As written in the ___DB Preparation Guide___, you should create a user with just read access for this. '''Encoding:''' The encoding of your database. If you’re not sure, leave the default, you can change it if you run into problems later. If you know your database is using a different encoding than latin-1 (which is typically the default for western languages), maybe UTF-8, select the appropriate entry in the drop-down list. Once you’ve entered the correct values, press <small>Save</small>. Click <small>Overview</small> in the link list at the top of the page to return to the datasource configuration page. If the connection status doesn’t turn into <small>OK</small> (and doesn’t after you’ve returned to the overview page), please read the __Debug guide__ about how to trace the problem. == Setting up the Database Structure == Clicking on “Edit DB Structure” on the Datasource Overview page will show the DB structure setup. Make sure the connection status at the top of the page still is OK. In a first step, you will add all tables/views that hold information you want to publish with BioCASe. You can ignore all database tables storing only data you don’t plan to map to ABCD. Same holds true if you use cache tables (as explained in the ___DB preparation guide___): Only the cache tables need to be set up on the database structure page. In a second step, you will specify the foreign key references between these tables. === Adding Tables/Views with their Primary Keys To add a table or view, just select it in the drop-down list. The name of the table will be filled in the Alias text box (you will use this alias later in the mapping process), feel free to change it into something that describes the contents of that tables (for example specimen for the root specimen table). Press Add to insert the table into the list. Before continuing to the next table, select the primary key in the list of attributes under Primary Key Atrribute(s). In case the data type is a string and not an integer, select text from the drop-down list. Press Add to add the primary key. If the primary key consists of several columns, you can add several primary key attributes. Redo this step (adding the tables first, then the primary key(s)) for all tables you want to publish data from. Always remember to set the correct primary key column(s), even if they’re not defined as primary keys in the database. Tables without a primary key cannot be used by the Provider Software and will be removed upon saving! === Specifying Foreign Key References === Once you’ve added all tables, you can define the relations between these tables by defining foreign keys. In order to do this, go to the Foreign Key(s) column of the table that holds a foreign key reference to another table, select the alias of the referenced table and press Add. Select the attribute that stores the foreign key and change the data type if it is not integer, then press Add. You must specify as many attributes for the foreign keys as the primary key of the referenced table has, otherwise the Provider Software won’t be able to create a valid join between the tables. Also, they must be in the same order and have identical data types. Redo this for all tables until you’ve specified all foreign key references. If you’ve installed the Graphviz package, the Provider Software will create a graph visualizing the current database structure. Each table will be visible as a node with the primary key(s) listed. References between tables will be represented by arrows, the foreign key(s) listed in the referring table: === Saving, Editing an Existing Setup === You can change aliases of tables you’ve already added to the list. To do so, go to the alias text box of that table and type in a different name. Upon exiting the text box, all references to this alias will be updated in the table setup; also the graph will reflect the changes. But caution: Schema mappings use these aliases to bind schema elements to tables and columns, if you change an alias you’ve already used in a schema mapping, you will break this binding! So it is advisable to decide for meaningful alias names and keep them stable once you’ve started the mapping process. Once you’ve added a table/view to the table list, you can’t change the underlying table, since this would invalidate all the primary and foreign key attributes. If you’ve added the wrong table, remove it by using the trash can symbol next to “Delete Alias” and add the correct table/view. If you’ve finished the table setup and the graph displays the model of your database correctly, save the configuration by Pressing Save at the bottom of the page. This is important; navigating away from the page without saving will cause your changes to be lost (even though you will be asked for confirmation)! It is advisable to save the table setup every now and then, for example once you’ve added all tables and primary keys. If you edit an existing table setup and happen to realize you’ve messed it up, you can revert to the state when you’ve pressed Save for the last time by using the Revert button at the bottom. But be careful: All unsaved changes will be lost. Note: The retrieval of tables/view/column names is currently only supported for the following DBMS: Microsoft SQL Server, MySQL, Access, Postgres and FoxPro. If you use a different DBMS, you will see a simple text box instead of a drop-down-list, where you’ll have to type in the name of the table or column you want to add. 3e6fd30cdc93604444b55b575c4d05f66be9ea1a 69 66 2011-06-23T14:35:23Z JoergHoletschek 3 /* Setting up the database connection */ wikitext text/x-wiki This tutorial explains in detail how to set up a datasource. == A BioCASe Data Source == Publishing a database with BioCASe means it will be made accessible through a BioCASe web service, which will be used later by the biodiversity network(s) you want to join to access to your data. For example, the GBIF Harvester will send a BioCASe Scan request to that web service to get a list of all scientific names kept in the database, and subsequently several BioCASe Search requests to load all records for these names. For each database you want to publish, you need to set up a BioCASe data source (do not mix that up with ODBC data sources on Windows machines). The resulting BioCASe web service is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source. So if you made your installation available at http://www.foobar.org/biocase during the installation process and set up a data source named Herbar, the URL of the BioCASe web service would be http://www.foobar.org/biocase/pywrapper.cgi?dsa=Herbar (data source names can be case sensitive, depending on your server’s operating system). == Creating a New Data Source == The start page of the BPS will always list the data sources currently configured, that means the web services offered by that BioCASe installation. On a vanilla installation, it will be empty: [[File:instBiocaseStartpage.png|500px]] Creating a new data source can be done in the configuration tool. Choose <small>Config Tool</small> on the start page, then <small>System Administration</small>. Log in with the password you set during the installation process (if you didn’t change that yet, the default is ''ACDC''). In the configuration screen, the sections <small>Data Sources</small> and <small>Recycle Bin</small> can be used to create, remove und un-remove a data source: [[File:dsaCreateDatasourceEmpty.png|500px]] For now, we want to create a new, empty data source. Enter a name – without any spaces – in the text box and click <small>Create DSA</small> (remember that the datasource name is case sensitive on Linux machines). After a second, it will appear in the list of existing data sources with a clickable link. By doing this, you created the first BioCASe web service for your installation. Clicking <small>Remove</small> will uninstall this service again (putting it in the recycle bin, so you can restore it later). Tip: If you select <small>abcdmetadata</small> in the template drop-down list, your data source will be created with an ABCD mapping for all the metadata elements stored in the metadata table mentioned in the ___Database preparation___ guide. So if you did use the templates in ''sampledb/abcdmetadata'' to create the metadata table in your database, you can use this template to spare the hassle of mapping this table and start right away with your occurrence data. == Setting up the database connection == Clicking on the data source you’ve just created will take you to the datasource configuration tool. Note that the <small>Schemas</small> section will be empty if you’ve created an empty data source or list two schemas (ABCD 1.2 and 2.06) if you created the datasource based on the <small>abcdmetadata</small> template: [[File:dsaOverview.png|450px]] The link in the <small>Database Connection</small> section will take you to the connection parameters dialog. Fill in the values for your database, press <small>Save</small> and hope for the connection to turn into a nice green <small>OK</small>: '''DBMS:''' Select the entry appropriate for your collection. The BPS can directly connect to database servers (MySQL, Postgres, DB2, SQL Server, and Oracle) and to Desktop-DBMS like Access and FoxPro through ODBC. ODBC is only available on Windows, so if you plan to use it, BioCASe must be installed on a Windows Server. '''Host:''' Enter the IP address of the database server. If you’re connection through ODBC, you must leave this empty. Instead, use Windows’ ODBC Administrator to set up a '''System datasource''' (!); the server or file name of the database will be stored in the ODBC configuration. '''Database:''' Enter the name of the ODBC datasource if you’re using ODBC, the name of the database if you’re directly connecting to a server. '''User/Password:''' Database credentials to be used. As written in the ___DB Preparation Guide___, you should create a user with just read access for this. '''Encoding:''' The encoding of your database. If you’re not sure, leave the default, you can change it if you run into problems later. If you know your database is using a different encoding than latin-1 (which is typically the default for western languages), maybe UTF-8, select the appropriate entry in the drop-down list. [[File:dsaDBConnection.png|450px]] Once you’ve entered the correct values, press <small>Save</small>. Click <small>Overview</small> in the link list at the top of the page to return to the datasource configuration page. If the connection status doesn’t turn into <small>OK</small> (and doesn’t after you’ve returned to the overview page), please read the __Debug guide__ about how to trace the problem. == Setting up the Database Structure == Clicking on “Edit DB Structure” on the Datasource Overview page will show the DB structure setup. Make sure the connection status at the top of the page still is OK. In a first step, you will add all tables/views that hold information you want to publish with BioCASe. You can ignore all database tables storing only data you don’t plan to map to ABCD. Same holds true if you use cache tables (as explained in the ___DB preparation guide___): Only the cache tables need to be set up on the database structure page. In a second step, you will specify the foreign key references between these tables. === Adding Tables/Views with their Primary Keys To add a table or view, just select it in the drop-down list. The name of the table will be filled in the Alias text box (you will use this alias later in the mapping process), feel free to change it into something that describes the contents of that tables (for example specimen for the root specimen table). Press Add to insert the table into the list. Before continuing to the next table, select the primary key in the list of attributes under Primary Key Atrribute(s). In case the data type is a string and not an integer, select text from the drop-down list. Press Add to add the primary key. If the primary key consists of several columns, you can add several primary key attributes. Redo this step (adding the tables first, then the primary key(s)) for all tables you want to publish data from. Always remember to set the correct primary key column(s), even if they’re not defined as primary keys in the database. Tables without a primary key cannot be used by the Provider Software and will be removed upon saving! === Specifying Foreign Key References === Once you’ve added all tables, you can define the relations between these tables by defining foreign keys. In order to do this, go to the Foreign Key(s) column of the table that holds a foreign key reference to another table, select the alias of the referenced table and press Add. Select the attribute that stores the foreign key and change the data type if it is not integer, then press Add. You must specify as many attributes for the foreign keys as the primary key of the referenced table has, otherwise the Provider Software won’t be able to create a valid join between the tables. Also, they must be in the same order and have identical data types. Redo this for all tables until you’ve specified all foreign key references. If you’ve installed the Graphviz package, the Provider Software will create a graph visualizing the current database structure. Each table will be visible as a node with the primary key(s) listed. References between tables will be represented by arrows, the foreign key(s) listed in the referring table: === Saving, Editing an Existing Setup === You can change aliases of tables you’ve already added to the list. To do so, go to the alias text box of that table and type in a different name. Upon exiting the text box, all references to this alias will be updated in the table setup; also the graph will reflect the changes. But caution: Schema mappings use these aliases to bind schema elements to tables and columns, if you change an alias you’ve already used in a schema mapping, you will break this binding! So it is advisable to decide for meaningful alias names and keep them stable once you’ve started the mapping process. Once you’ve added a table/view to the table list, you can’t change the underlying table, since this would invalidate all the primary and foreign key attributes. If you’ve added the wrong table, remove it by using the trash can symbol next to “Delete Alias” and add the correct table/view. If you’ve finished the table setup and the graph displays the model of your database correctly, save the configuration by Pressing Save at the bottom of the page. This is important; navigating away from the page without saving will cause your changes to be lost (even though you will be asked for confirmation)! It is advisable to save the table setup every now and then, for example once you’ve added all tables and primary keys. If you edit an existing table setup and happen to realize you’ve messed it up, you can revert to the state when you’ve pressed Save for the last time by using the Revert button at the bottom. But be careful: All unsaved changes will be lost. Note: The retrieval of tables/view/column names is currently only supported for the following DBMS: Microsoft SQL Server, MySQL, Access, Postgres and FoxPro. If you use a different DBMS, you will see a simple text box instead of a drop-down-list, where you’ll have to type in the name of the table or column you want to add. b7ae46ed78c132a1cb473b4266655f09bf3a8249 70 69 2011-06-23T14:44:58Z JoergHoletschek 3 /* Setting up the Database Structure */ wikitext text/x-wiki This tutorial explains in detail how to set up a datasource. == A BioCASe Data Source == Publishing a database with BioCASe means it will be made accessible through a BioCASe web service, which will be used later by the biodiversity network(s) you want to join to access to your data. For example, the GBIF Harvester will send a BioCASe Scan request to that web service to get a list of all scientific names kept in the database, and subsequently several BioCASe Search requests to load all records for these names. For each database you want to publish, you need to set up a BioCASe data source (do not mix that up with ODBC data sources on Windows machines). The resulting BioCASe web service is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source. So if you made your installation available at http://www.foobar.org/biocase during the installation process and set up a data source named Herbar, the URL of the BioCASe web service would be http://www.foobar.org/biocase/pywrapper.cgi?dsa=Herbar (data source names can be case sensitive, depending on your server’s operating system). == Creating a New Data Source == The start page of the BPS will always list the data sources currently configured, that means the web services offered by that BioCASe installation. On a vanilla installation, it will be empty: [[File:instBiocaseStartpage.png|500px]] Creating a new data source can be done in the configuration tool. Choose <small>Config Tool</small> on the start page, then <small>System Administration</small>. Log in with the password you set during the installation process (if you didn’t change that yet, the default is ''ACDC''). In the configuration screen, the sections <small>Data Sources</small> and <small>Recycle Bin</small> can be used to create, remove und un-remove a data source: [[File:dsaCreateDatasourceEmpty.png|500px]] For now, we want to create a new, empty data source. Enter a name – without any spaces – in the text box and click <small>Create DSA</small> (remember that the datasource name is case sensitive on Linux machines). After a second, it will appear in the list of existing data sources with a clickable link. By doing this, you created the first BioCASe web service for your installation. Clicking <small>Remove</small> will uninstall this service again (putting it in the recycle bin, so you can restore it later). Tip: If you select <small>abcdmetadata</small> in the template drop-down list, your data source will be created with an ABCD mapping for all the metadata elements stored in the metadata table mentioned in the ___Database preparation___ guide. So if you did use the templates in ''sampledb/abcdmetadata'' to create the metadata table in your database, you can use this template to spare the hassle of mapping this table and start right away with your occurrence data. == Setting up the database connection == Clicking on the data source you’ve just created will take you to the datasource configuration tool. Note that the <small>Schemas</small> section will be empty if you’ve created an empty data source or list two schemas (ABCD 1.2 and 2.06) if you created the datasource based on the <small>abcdmetadata</small> template: [[File:dsaOverview.png|450px]] The link in the <small>Database Connection</small> section will take you to the connection parameters dialog. Fill in the values for your database, press <small>Save</small> and hope for the connection to turn into a nice green <small>OK</small>: '''DBMS:''' Select the entry appropriate for your collection. The BPS can directly connect to database servers (MySQL, Postgres, DB2, SQL Server, and Oracle) and to Desktop-DBMS like Access and FoxPro through ODBC. ODBC is only available on Windows, so if you plan to use it, BioCASe must be installed on a Windows Server. '''Host:''' Enter the IP address of the database server. If you’re connection through ODBC, you must leave this empty. Instead, use Windows’ ODBC Administrator to set up a '''System datasource''' (!); the server or file name of the database will be stored in the ODBC configuration. '''Database:''' Enter the name of the ODBC datasource if you’re using ODBC, the name of the database if you’re directly connecting to a server. '''User/Password:''' Database credentials to be used. As written in the ___DB Preparation Guide___, you should create a user with just read access for this. '''Encoding:''' The encoding of your database. If you’re not sure, leave the default, you can change it if you run into problems later. If you know your database is using a different encoding than latin-1 (which is typically the default for western languages), maybe UTF-8, select the appropriate entry in the drop-down list. [[File:dsaDBConnection.png|450px]] Once you’ve entered the correct values, press <small>Save</small>. Click <small>Overview</small> in the link list at the top of the page to return to the datasource configuration page. If the connection status doesn’t turn into <small>OK</small> (and doesn’t after you’ve returned to the overview page), please read the __Debug guide__ about how to trace the problem. == Setting up the Database Structure == Clicking on <small>Edit DB Structure</small> on the datasource overview page will show the DB structure setup. Make sure the connection status at the top of the page still is <small>OK</small>. In a first step, you will add all tables/views that hold information you want to publish with BioCASe. You can ignore all database tables storing only data you don’t plan to map to ABCD. Same holds true if you use cache tables (as explained in the ___DB preparation guide___): Only the cache tables need to be set up on the database structure page. In a second step, you will specify the foreign key references between these tables. === Adding Tables/Views with their Primary Keys === To add a table or view, just select it in the drop-down list. The name of the table will be filled in the <small>Alias</small> text box (you will use this alias later in the mapping process), feel free to change it into something that describes the contents of that tables (for example ''specimen'' for the root specimen table). Press <small>Add</small> to insert the table into the list. Before continuing to the next table, select the primary key in the list of attributes under <small>Primary Key Attribute(s)</small>. In case the data type is a string and not an integer, select <small>text</small> from the drop-down list. Press <small>Add</small> to add the primary key. If the primary key consists of several columns, you can add several primary key attributes. Redo this step (adding the tables first, then the primary key(s)) for all tables you want to publish data from. Always remember to set the correct primary key column(s), even if they’re not defined as primary keys in the database. Tables without a primary key cannot be used by the Provider Software and will be removed upon saving! === Specifying Foreign Key References === Once you’ve added all tables, you can define the relations between these tables by defining foreign keys. In order to do this, go to the <small>Foreign Key(s)</small> column of the table that holds a foreign key reference to another table, select the alias of the referenced table and press <small>Add</small>. Select the attribute that stores the foreign key and change the data type if it is not integer, then press <small>Add</small>. You must specify as many attributes for the foreign keys as the primary key of the referenced table has, otherwise the Provider Software won’t be able to create a valid join between the tables. Also, they must be in the same order and have identical data types. Redo this for all tables until you’ve specified all foreign key references. If you’ve installed the Graphviz package, the Provider Software will create a graph visualizing the current database structure. Each table will be visible as a node with the primary key(s) listed. References between tables will be represented by arrows, the foreign key(s) listed in the referring table: [[File:dsaDatabaseStructure_AlgenEngels.png]] === Saving, Editing an Existing Setup === You can change aliases of tables you’ve already added to the list. To do so, go to the <small>Alias</small> text box of that table and type in a different name. Upon exiting the text box, all references to this alias will be updated in the table setup; also the graph will reflect the changes. But caution: Schema mappings use these aliases to bind schema elements to tables and columns, if you change an alias you’ve already used in a schema mapping, you will break this binding! So it is advisable to decide for meaningful alias names and keep them stable once you’ve started the mapping process. Once you’ve added a table/view to the table list, you can’t change the underlying table, since this would invalidate all the primary and foreign key attributes. If you’ve added the wrong table, remove it by using the trash can symbol next to <small>Delete Alias</small> and add the correct table/view. If you’ve finished the table setup and the graph displays the model of your database correctly, save the configuration by pressing <small>Save</small> at the bottom of the page. This is important; navigating away from the page without saving will cause your changes to be lost (even though you will be asked for confirmation)! It is advisable to save the table setup every now and then, for example once you’ve added all tables and primary keys. If you edit an existing table setup and happen to realize you’ve messed it up, you can revert to the state when you’ve pressed Save for the last time by using the <small>Revert</small> button at the bottom. But be careful: All unsaved changes will be lost. Note: The retrieval of tables/view/column names is currently only supported for the following DBMS: Microsoft SQL Server, MySQL, Access, Postgres and FoxPro. If you use a different DBMS, you will see a simple text box instead of a drop-down-list, where you’ll have to type in the name of the table or column you want to add. 6553163b8defd2513d7aaeb7ebac647823e6744d 72 70 2011-06-23T14:45:57Z JoergHoletschek 3 /* Setting up the Database Structure */ wikitext text/x-wiki This tutorial explains in detail how to set up a datasource. == A BioCASe Data Source == Publishing a database with BioCASe means it will be made accessible through a BioCASe web service, which will be used later by the biodiversity network(s) you want to join to access to your data. For example, the GBIF Harvester will send a BioCASe Scan request to that web service to get a list of all scientific names kept in the database, and subsequently several BioCASe Search requests to load all records for these names. For each database you want to publish, you need to set up a BioCASe data source (do not mix that up with ODBC data sources on Windows machines). The resulting BioCASe web service is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source. So if you made your installation available at http://www.foobar.org/biocase during the installation process and set up a data source named Herbar, the URL of the BioCASe web service would be http://www.foobar.org/biocase/pywrapper.cgi?dsa=Herbar (data source names can be case sensitive, depending on your server’s operating system). == Creating a New Data Source == The start page of the BPS will always list the data sources currently configured, that means the web services offered by that BioCASe installation. On a vanilla installation, it will be empty: [[File:instBiocaseStartpage.png|500px]] Creating a new data source can be done in the configuration tool. Choose <small>Config Tool</small> on the start page, then <small>System Administration</small>. Log in with the password you set during the installation process (if you didn’t change that yet, the default is ''ACDC''). In the configuration screen, the sections <small>Data Sources</small> and <small>Recycle Bin</small> can be used to create, remove und un-remove a data source: [[File:dsaCreateDatasourceEmpty.png|500px]] For now, we want to create a new, empty data source. Enter a name – without any spaces – in the text box and click <small>Create DSA</small> (remember that the datasource name is case sensitive on Linux machines). After a second, it will appear in the list of existing data sources with a clickable link. By doing this, you created the first BioCASe web service for your installation. Clicking <small>Remove</small> will uninstall this service again (putting it in the recycle bin, so you can restore it later). Tip: If you select <small>abcdmetadata</small> in the template drop-down list, your data source will be created with an ABCD mapping for all the metadata elements stored in the metadata table mentioned in the ___Database preparation___ guide. So if you did use the templates in ''sampledb/abcdmetadata'' to create the metadata table in your database, you can use this template to spare the hassle of mapping this table and start right away with your occurrence data. == Setting up the database connection == Clicking on the data source you’ve just created will take you to the datasource configuration tool. Note that the <small>Schemas</small> section will be empty if you’ve created an empty data source or list two schemas (ABCD 1.2 and 2.06) if you created the datasource based on the <small>abcdmetadata</small> template: [[File:dsaOverview.png|450px]] The link in the <small>Database Connection</small> section will take you to the connection parameters dialog. Fill in the values for your database, press <small>Save</small> and hope for the connection to turn into a nice green <small>OK</small>: '''DBMS:''' Select the entry appropriate for your collection. The BPS can directly connect to database servers (MySQL, Postgres, DB2, SQL Server, and Oracle) and to Desktop-DBMS like Access and FoxPro through ODBC. ODBC is only available on Windows, so if you plan to use it, BioCASe must be installed on a Windows Server. '''Host:''' Enter the IP address of the database server. If you’re connection through ODBC, you must leave this empty. Instead, use Windows’ ODBC Administrator to set up a '''System datasource''' (!); the server or file name of the database will be stored in the ODBC configuration. '''Database:''' Enter the name of the ODBC datasource if you’re using ODBC, the name of the database if you’re directly connecting to a server. '''User/Password:''' Database credentials to be used. As written in the ___DB Preparation Guide___, you should create a user with just read access for this. '''Encoding:''' The encoding of your database. If you’re not sure, leave the default, you can change it if you run into problems later. If you know your database is using a different encoding than latin-1 (which is typically the default for western languages), maybe UTF-8, select the appropriate entry in the drop-down list. [[File:dsaDBConnection.png|450px]] Once you’ve entered the correct values, press <small>Save</small>. Click <small>Overview</small> in the link list at the top of the page to return to the datasource configuration page. If the connection status doesn’t turn into <small>OK</small> (and doesn’t after you’ve returned to the overview page), please read the __Debug guide__ about how to trace the problem. == Setting up the Database Structure == Clicking on <small>Edit DB Structure</small> on the datasource overview page will show the DB structure setup. Make sure the connection status at the top of the page still is <small>OK</small>. In a first step, you will add all tables/views that hold information you want to publish with BioCASe. You can ignore all database tables storing only data you don’t plan to map to ABCD. Same holds true if you use cache tables (as explained in the ___DB preparation guide___): Only the cache tables need to be set up on the database structure page. In a second step, you will specify the foreign key references between these tables. === Adding Tables/Views with their Primary Keys === To add a table or view, just select it in the drop-down list. The name of the table will be filled in the <small>Alias</small> text box (you will use this alias later in the mapping process), feel free to change it into something that describes the contents of that tables (for example ''specimen'' for the root specimen table). Press <small>Add</small> to insert the table into the list. Before continuing to the next table, select the primary key in the list of attributes under <small>Primary Key Attribute(s)</small>. In case the data type is a string and not an integer, select <small>text</small> from the drop-down list. Press <small>Add</small> to add the primary key. If the primary key consists of several columns, you can add several primary key attributes. Redo this step (adding the tables first, then the primary key(s)) for all tables you want to publish data from. Always remember to set the correct primary key column(s), even if they’re not defined as primary keys in the database. Tables without a primary key cannot be used by the Provider Software and will be removed upon saving! === Specifying Foreign Key References === Once you’ve added all tables, you can define the relations between these tables by defining foreign keys. In order to do this, go to the <small>Foreign Key(s)</small> column of the table that holds a foreign key reference to another table, select the alias of the referenced table and press <small>Add</small>. Select the attribute that stores the foreign key and change the data type if it is not integer, then press <small>Add</small>. You must specify as many attributes for the foreign keys as the primary key of the referenced table has, otherwise the Provider Software won’t be able to create a valid join between the tables. Also, they must be in the same order and have identical data types. Redo this for all tables until you’ve specified all foreign key references. If you’ve installed the Graphviz package, the Provider Software will create a graph visualizing the current database structure. Each table will be visible as a node with the primary key(s) listed. References between tables will be represented by arrows, the foreign key(s) listed in the referring table: [[File:dsaDatabaseStructure_AlgenEngels.png|800px]] === Saving, Editing an Existing Setup === You can change aliases of tables you’ve already added to the list. To do so, go to the <small>Alias</small> text box of that table and type in a different name. Upon exiting the text box, all references to this alias will be updated in the table setup; also the graph will reflect the changes. But caution: Schema mappings use these aliases to bind schema elements to tables and columns, if you change an alias you’ve already used in a schema mapping, you will break this binding! So it is advisable to decide for meaningful alias names and keep them stable once you’ve started the mapping process. Once you’ve added a table/view to the table list, you can’t change the underlying table, since this would invalidate all the primary and foreign key attributes. If you’ve added the wrong table, remove it by using the trash can symbol next to <small>Delete Alias</small> and add the correct table/view. If you’ve finished the table setup and the graph displays the model of your database correctly, save the configuration by pressing <small>Save</small> at the bottom of the page. This is important; navigating away from the page without saving will cause your changes to be lost (even though you will be asked for confirmation)! It is advisable to save the table setup every now and then, for example once you’ve added all tables and primary keys. If you edit an existing table setup and happen to realize you’ve messed it up, you can revert to the state when you’ve pressed Save for the last time by using the <small>Revert</small> button at the bottom. But be careful: All unsaved changes will be lost. Note: The retrieval of tables/view/column names is currently only supported for the following DBMS: Microsoft SQL Server, MySQL, Access, Postgres and FoxPro. If you use a different DBMS, you will see a simple text box instead of a drop-down-list, where you’ll have to type in the name of the table or column you want to add. fd9199b389217fe491709dc342dbcfb0c8f6096e 73 72 2011-06-23T14:49:13Z JoergHoletschek 3 /* Saving, Editing an Existing Setup */ wikitext text/x-wiki This tutorial explains in detail how to set up a datasource. == A BioCASe Data Source == Publishing a database with BioCASe means it will be made accessible through a BioCASe web service, which will be used later by the biodiversity network(s) you want to join to access to your data. For example, the GBIF Harvester will send a BioCASe Scan request to that web service to get a list of all scientific names kept in the database, and subsequently several BioCASe Search requests to load all records for these names. For each database you want to publish, you need to set up a BioCASe data source (do not mix that up with ODBC data sources on Windows machines). The resulting BioCASe web service is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source. So if you made your installation available at http://www.foobar.org/biocase during the installation process and set up a data source named Herbar, the URL of the BioCASe web service would be http://www.foobar.org/biocase/pywrapper.cgi?dsa=Herbar (data source names can be case sensitive, depending on your server’s operating system). == Creating a New Data Source == The start page of the BPS will always list the data sources currently configured, that means the web services offered by that BioCASe installation. On a vanilla installation, it will be empty: [[File:instBiocaseStartpage.png|500px]] Creating a new data source can be done in the configuration tool. Choose <small>Config Tool</small> on the start page, then <small>System Administration</small>. Log in with the password you set during the installation process (if you didn’t change that yet, the default is ''ACDC''). In the configuration screen, the sections <small>Data Sources</small> and <small>Recycle Bin</small> can be used to create, remove und un-remove a data source: [[File:dsaCreateDatasourceEmpty.png|500px]] For now, we want to create a new, empty data source. Enter a name – without any spaces – in the text box and click <small>Create DSA</small> (remember that the datasource name is case sensitive on Linux machines). After a second, it will appear in the list of existing data sources with a clickable link. By doing this, you created the first BioCASe web service for your installation. Clicking <small>Remove</small> will uninstall this service again (putting it in the recycle bin, so you can restore it later). Tip: If you select <small>abcdmetadata</small> in the template drop-down list, your data source will be created with an ABCD mapping for all the metadata elements stored in the metadata table mentioned in the ___Database preparation___ guide. So if you did use the templates in ''sampledb/abcdmetadata'' to create the metadata table in your database, you can use this template to spare the hassle of mapping this table and start right away with your occurrence data. == Setting up the database connection == Clicking on the data source you’ve just created will take you to the datasource configuration tool. Note that the <small>Schemas</small> section will be empty if you’ve created an empty data source or list two schemas (ABCD 1.2 and 2.06) if you created the datasource based on the <small>abcdmetadata</small> template: [[File:dsaOverview.png|450px]] The link in the <small>Database Connection</small> section will take you to the connection parameters dialog. Fill in the values for your database, press <small>Save</small> and hope for the connection to turn into a nice green <small>OK</small>: '''DBMS:''' Select the entry appropriate for your collection. The BPS can directly connect to database servers (MySQL, Postgres, DB2, SQL Server, and Oracle) and to Desktop-DBMS like Access and FoxPro through ODBC. ODBC is only available on Windows, so if you plan to use it, BioCASe must be installed on a Windows Server. '''Host:''' Enter the IP address of the database server. If you’re connection through ODBC, you must leave this empty. Instead, use Windows’ ODBC Administrator to set up a '''System datasource''' (!); the server or file name of the database will be stored in the ODBC configuration. '''Database:''' Enter the name of the ODBC datasource if you’re using ODBC, the name of the database if you’re directly connecting to a server. '''User/Password:''' Database credentials to be used. As written in the ___DB Preparation Guide___, you should create a user with just read access for this. '''Encoding:''' The encoding of your database. If you’re not sure, leave the default, you can change it if you run into problems later. If you know your database is using a different encoding than latin-1 (which is typically the default for western languages), maybe UTF-8, select the appropriate entry in the drop-down list. [[File:dsaDBConnection.png|450px]] Once you’ve entered the correct values, press <small>Save</small>. Click <small>Overview</small> in the link list at the top of the page to return to the datasource configuration page. If the connection status doesn’t turn into <small>OK</small> (and doesn’t after you’ve returned to the overview page), please read the __Debug guide__ about how to trace the problem. == Setting up the Database Structure == Clicking on <small>Edit DB Structure</small> on the datasource overview page will show the DB structure setup. Make sure the connection status at the top of the page still is <small>OK</small>. In a first step, you will add all tables/views that hold information you want to publish with BioCASe. You can ignore all database tables storing only data you don’t plan to map to ABCD. Same holds true if you use cache tables (as explained in the ___DB preparation guide___): Only the cache tables need to be set up on the database structure page. In a second step, you will specify the foreign key references between these tables. === Adding Tables/Views with their Primary Keys === To add a table or view, just select it in the drop-down list. The name of the table will be filled in the <small>Alias</small> text box (you will use this alias later in the mapping process), feel free to change it into something that describes the contents of that tables (for example ''specimen'' for the root specimen table). Press <small>Add</small> to insert the table into the list. Before continuing to the next table, select the primary key in the list of attributes under <small>Primary Key Attribute(s)</small>. In case the data type is a string and not an integer, select <small>text</small> from the drop-down list. Press <small>Add</small> to add the primary key. If the primary key consists of several columns, you can add several primary key attributes. Redo this step (adding the tables first, then the primary key(s)) for all tables you want to publish data from. Always remember to set the correct primary key column(s), even if they’re not defined as primary keys in the database. Tables without a primary key cannot be used by the Provider Software and will be removed upon saving! === Specifying Foreign Key References === Once you’ve added all tables, you can define the relations between these tables by defining foreign keys. In order to do this, go to the <small>Foreign Key(s)</small> column of the table that holds a foreign key reference to another table, select the alias of the referenced table and press <small>Add</small>. Select the attribute that stores the foreign key and change the data type if it is not integer, then press <small>Add</small>. You must specify as many attributes for the foreign keys as the primary key of the referenced table has, otherwise the Provider Software won’t be able to create a valid join between the tables. Also, they must be in the same order and have identical data types. Redo this for all tables until you’ve specified all foreign key references. If you’ve installed the Graphviz package, the Provider Software will create a graph visualizing the current database structure. Each table will be visible as a node with the primary key(s) listed. References between tables will be represented by arrows, the foreign key(s) listed in the referring table: [[File:dsaDatabaseStructure_AlgenEngels.png|800px]] === Saving, Editing an Existing Setup === You can change aliases of tables you’ve already added to the list. To do so, go to the <small>Alias</small> text box of that table and type in a different name. Upon exiting the text box, all references to this alias will be updated in the table setup; also the graph will reflect the changes. But caution: Schema mappings use these aliases to bind schema elements to tables and columns, if you change an alias you’ve already used in a schema mapping, you will break this binding! So it is advisable to decide for meaningful alias names and keep them stable once you’ve started the mapping process. Once you’ve added a table/view to the table list, you can’t change the underlying table, since this would invalidate all the primary and foreign key attributes. If you’ve added the wrong table, remove it by using the trash can symbol next to <small>Delete Alias</small> and add the correct table/view. If you’ve finished the table setup and the graph displays the model of your database correctly, save the configuration by pressing <small>Save</small> at the bottom of the page. This is important; navigating away from the page without saving will cause your changes to be lost (even though you will be asked for confirmation)! It is advisable to save the table setup every now and then, for example once you’ve added all tables and primary keys. If you edit an existing table setup and happen to realize you’ve messed it up, you can revert to the state when you’ve pressed Save for the last time by using the <small>Revert</small> button at the bottom. But be careful: All unsaved changes will be lost. '''<u>Important Note:</u>''' The retrieval of tables/view/column names is currently only supported for the following DBMS: Microsoft SQL Server, MySQL, Access, Postgres and FoxPro. If you use a different DBMS, you will see a simple text box instead of a drop-down-list, where you’ll have to type in the name of the table or column you want to add. 29e8fc31176b1be43fe716898b9ae855e1495492 6 25 64 2011-06-23T14:20:49Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 26 67 2011-06-23T14:32:56Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 27 68 2011-06-23T14:33:29Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 28 71 2011-06-23T14:45:09Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 0 29 74 2011-06-30T10:15:18Z JoergHoletschek 3 Created page with "This tutorial will explain how to set up a mapping for the ABCD2.06 data schema. If you intend to map another schema (e.g. one of the ABCD extension ABCD-EFG, ABCD-DNA or HISPID)..." wikitext text/x-wiki This tutorial will explain how to set up a mapping for the ABCD2.06 data schema. If you intend to map another schema (e.g. one of the ABCD extension ABCD-EFG, ABCD-DNA or HISPID), you still should take the time to read it. Large portions of the process will be identical. == Access to Biological Collection Data Schema (ABCD) == ABCD in its current version 2.06 is the data schema typically used in conjunction with BioCASe. It is a highly complex XML data schema with about 1,000 elements that is able to store almost every piece of information that can be found in a natural history specimen collection or an observation database. The complexity is due to the fact that it can be used for a wide range of collections/databases – for living and preserved specimens, observations and culture collections, for zoological, botanical, bacterial and viral collections, marine or terrestrial, for herbaria, botanic and zoological gardens. For each of these special types, ABCD features special sections where information specific for this type can be stored. Other sections of ABCD will be shared by all these types, for example for gathering site, identifier or metadata. The full documentation on ABCD can be found here, a list of commonly used ABCD elements here. == Creating an ABCD Mapping for a Data Source == If you created a data source from a template (e.g. ‘’abcdmetadata’’), a schema mapping for ABCD will be already existent. In this case, you can directly jump to __Mapping editor__ section. If you created an empty data source, the section <small>Schemas</small>in the data source configuration overview will be empty: In order to create a new schema mapping, select the desired schema from the list and click <small>create</small>. For ABCD, choose <small>ABCD_2.06.xml</small> (be careful not to hit one of the ABCD extensions, they look quite similar). After clicking <small>create</small>, you will be directed to the mapping editor, which will allow you to edit the mappings. == The Mapping Editor == The mapping editor allows you to add, remove and edit ABCD mappings. On top you’ll find some summary information, on the bottom you’ll see the structure of the ABCD mapping: First thing we will do is to set the root table, which is the table that holds the records you want to publish (specimens or observations). That table must hold one record per object to be published, identified by a unique identifier (called ‘’UnitID’’). This identifier must not be the primary key, as long as it’s unique (if that sentence confuses you, ignore it and just keep in mind that the UnitID must be unique.) Use the drop down box labelled <small>Root table Alias</small> to select the root table of your data model. If your metadata table holds just one record (because all occurrences published share the same metadata) and if it is not linked to the root table by a foreign key, you can specify this table in the drop-down box <small>Static table alias</small>. The Provider Software will use a natural join to connect the metadata record stored in this table to all occurrences published. If your metadata table is referenced by a foreign key in the root able, you must not declare the metadata table as a static table! So either join the metadata table using a foreign key OR declare it as a static table. When you’ve set the root table (and static table, if you use that), press <small>Save</small>. As with the DB structure setup, this will write your changes to the configuration files, closing the tab without saving will discard your changes! Also, in case you messed something up, you can use the Revert button to restore the state when you’ve saved for the last time. In the mapping editor, the link Overview at the top of the page will take you back to the datasource configuration overview. After creating a mapping, it will appear in the section <small>Schemas</small>. The entry will show the number of mapped elements (0 for now) and the schema namespace. Pressing the trash can symbol will move it to the trash can, which means that your web service does not support this schema anymore; if you did this by mistake, you can restore it by pressing the <small>Restore</small> button. <small>Purge</small> will delete the schema mapping permanently: You can map several schemas for a given datasource. For example, in addition to the current ABCD2 standard, you could add the deprecated version 1.2 (even though I have no idea why you should do that) or DarwinCore. Just select the appropriate schema in the drop-down list and click <small>Create</small>. Since the namespace of a schema will be used in requests, you cannot create two mappings for the same namespace. Trying this will get you an error message. == Adding Mandatory ABCD Elements == Clicking on a schema in the datasource overview page will take you again to the mapping editor. In the lower half of the page, you will see the basic structure of an ABCD document: Technical/Content Contact: These trees hold information on who should be contacted for technical issues or questions concerning the published database, each with name, address, email address and telephone number. Metadata: In this sub tree you’ll find all metadata fields, for example title and description of the dataset, ownership information, date last modified, IPR statements etc. Units: This part stores the “real” data you want to publish, that means all specimen or observation data. The three elements already shown in the tree are identifiers that make up the so called “Triple ID”, which uniquely identifies records within primary biodiversity networks such as BioCASe and GBIF: - ‘’UnitID’’, which is a unique identifier for each record to be published (you can use the catalogue number, for example, or an existing barcode); - ‘’ SourceInstitutionID’’, an identifier for the institution publishing the dataset (for example BGBM); - ‘’ SourceID’’, an identifier for the dataset (e.g. “Herbarium Berolinense’’). The elements already shown in the mapping editor are printed in red because they are mandatory, which means they are required for an ABCD document. Without them being mapped (to database fields that are non-empty), the Provider Software will not be able to construct valid ABCD documents and will not publish any records. So the first elements that should be mapped are the mandatory fields. To add a mapping, click the <strong>+</strong> symbol next to the field, which will open the mapping editor dialog, where you can select the database table/column you want to map the ABCD element to: In this case, the element /DataSets/DataSet/Metadata/RevisionData/DateModified is mapped to the column ‘’source_update’’ of the ‘’metadata’’ table, which has the data type ‘’date’’. (Remember that table/column retrieval is not supported for all DBMS, so if you’re unlucky, you might need to type in table/column name manually.) <small>Save</small> will accept your settings and close the dialog, <small>Cancel</small> will discard any changes. Instead of mapping an element to a database field, you can map it to a literal. For example, you could enter the technical and content contact information directly as literals in the mapping editor dialog (assuming they’re the same for all records). However, we strongly discourage you from doing this. For one thing, this is just bad style, since it will scatter the information published over different places: Some will be retrieved from the database, some (the literals) will be loaded from the configuration files. Changing the metadata of your service later would then require access to the database and access to the BioCASe configuration – that means the configuration tool password. Be nice to the person who will do these changes and put everything neatly in a database table! The second reason is even more important: Using literals for an element will result in this element to become not searchable, which means it cannot be used in a filter of a Search or a Scan request. Even though this behaviour is discoverable with a Capabilities request (by evaluating the ‘’searchable’’ flag), a potential user of your web service might just not think of that, so it’s a good idea to avoid that in first place. Please use literals sparsely! In the mapping editor window, you can select several source columns and several literals, even a combination of both. The Provider Software will simply concatenate the values retrieved from the database and the literals specified (in the order they’re given). But again: Keep in mind that this will make the element not searchable. A better solution is to do any concatenation operations in a database view. Use the mapping editor window to add mappings for all eight mandatory fields: Click on the <strong>+</strong> symbol next to the element, choose table/column name and the correct data type, then press <small>Save</small>. When you’re done, the page should look similar to the following image. Press <small>Save</small> in the main window to write your changes to the configuration files. It should now look similar to this: == Initial Testing of the Data Source == Once you’ve mapped all mandatory elements, you can do the first test of your BioCASe web service. Even though you haven’t mapped any real information yet, it is a good idea to check the service and solve any problems hitherto before continuing with the mapping. Click on <small>Test mapping!</small> to open the Query Form that can be used for this purpose: With the Query Form you can send a BioCASe request to your web service; it will wait for the reply of the web service and display the response document. On top of the page you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ‘’flora’’ being the name of the datasource. Make sure this is correct for your installation! Into the text box below you can enter the request XML text. Because it would be cumbersome to enter the whole BioCASe request, you can use the links below the box to load templates. For now, click on “ABCD2 Search” to fill the box with a BioCASe Search request for documents in the ABCD2 format (the one we’ve just mapped): As you can see, there is a filter set on the ‘’ScientificName’’ element. Since we haven’t mapped this element yet, this won’t work. So let’s remove the filter (all lines between <filter> and </filter>), leaving the text box like this: Pressing <small>Submit</small> will send the request to the web service and – hopefully – display the response document with the correct ABCD records. For the mapping we’ve just created, it should look similar to the following document (use the – symbol next to the trees TechnicalContact, ContentContact and Metadata to collapse them): If you get an error, please check out the ___debugging tutorial__ on how to debug a web service. You should continue the mapping process only after your web service returns results for the basal mapping we’ve just created. == Adding additional ABCD Elements == Once you’ve successfully managed to map the mandatory ABCD elements (“successfully” means the web service returns ABCD documents), you can go on with the mapping process. This is usually an iterative process: You add a handful of elements (for example, country, country code, locality text and coordinates for the gathering site), then you use the Query Form to see if the information ends up where it is supposed to do. This is the reason the Query Form opens up in a separate tab (or window, of you configured your web browser so): In one tab, you use the mapping editor to add new elements; after pressing Save you go to the Query Form tab and see what the response documents look like. Redo this until you’ve mapped all information you want to publish. In order to map new ABCD concepts, tick the check box labelled <small>Show all concepts</small> and press <small>Refresh</small>. The mapping editor will then display the whole ABCD tree, which looks admittedly a bit intimidating. In order to find the concept you’re looking for, use the search function of your browser. If you want to map the country name of the gathering site, for example, hit <small>Ctrl + F</small> (for Firefox, Microsoft IE and Opera) and type in the name of the ABCD element, namely <small>country</small>. The browser will jump to the Country tree of the Gathering element, where you’ll find the elements for country name (Name), ISO country code (ISO3166Code) and the country name in the country’s language (NameDerived): Continue the sequence mapping new elements/testing until you’ve mapped all information you want to publish. You can have a look at the list of recommend and commonly used ABCD elements here. The full ABCD documentation can be found here. Here you can have a look at a sample response ABCD document for a web service with 77 concepts mapped. Of course your documents will look different, because you will probably use different ABCD elements. But still it’s good to get an idea: asdfads 58fd0f802d1e3b00efb2572b971147321fb2ec88 75 74 2011-06-30T10:18:38Z JoergHoletschek 3 /* Access to Biological Collection Data Schema (ABCD) */ wikitext text/x-wiki This tutorial will explain how to set up a mapping for the ABCD2.06 data schema. If you intend to map another schema (e.g. one of the ABCD extension ABCD-EFG, ABCD-DNA or HISPID), you still should take the time to read it. Large portions of the process will be identical. == Access to Biological Collection Data Schema (ABCD) == ABCD in its current version 2.06 is the data schema typically used in conjunction with BioCASe. It is a highly complex XML data schema with about 1,000 elements that is able to store almost every piece of information that can be found in a natural history specimen collection or an observation database. The complexity is due to the fact that it can be used for a wide range of collections/databases – for living and preserved specimens, observations and culture collections, for zoological, botanical, bacterial and viral collections, marine or terrestrial, for herbaria, botanic and zoological gardens. For each of these special types, ABCD features special sections where information specific for this type can be stored. Other sections of ABCD will be shared by all these types, for example for gathering site, identifier or metadata. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a list of commonly used ABCD elements on the Wiki page [[CommonABCD2Concepts]]. == Creating an ABCD Mapping for a Data Source == If you created a data source from a template (e.g. ‘’abcdmetadata’’), a schema mapping for ABCD will be already existent. In this case, you can directly jump to __Mapping editor__ section. If you created an empty data source, the section <small>Schemas</small>in the data source configuration overview will be empty: In order to create a new schema mapping, select the desired schema from the list and click <small>create</small>. For ABCD, choose <small>ABCD_2.06.xml</small> (be careful not to hit one of the ABCD extensions, they look quite similar). After clicking <small>create</small>, you will be directed to the mapping editor, which will allow you to edit the mappings. == The Mapping Editor == The mapping editor allows you to add, remove and edit ABCD mappings. On top you’ll find some summary information, on the bottom you’ll see the structure of the ABCD mapping: First thing we will do is to set the root table, which is the table that holds the records you want to publish (specimens or observations). That table must hold one record per object to be published, identified by a unique identifier (called ‘’UnitID’’). This identifier must not be the primary key, as long as it’s unique (if that sentence confuses you, ignore it and just keep in mind that the UnitID must be unique.) Use the drop down box labelled <small>Root table Alias</small> to select the root table of your data model. If your metadata table holds just one record (because all occurrences published share the same metadata) and if it is not linked to the root table by a foreign key, you can specify this table in the drop-down box <small>Static table alias</small>. The Provider Software will use a natural join to connect the metadata record stored in this table to all occurrences published. If your metadata table is referenced by a foreign key in the root able, you must not declare the metadata table as a static table! So either join the metadata table using a foreign key OR declare it as a static table. When you’ve set the root table (and static table, if you use that), press <small>Save</small>. As with the DB structure setup, this will write your changes to the configuration files, closing the tab without saving will discard your changes! Also, in case you messed something up, you can use the Revert button to restore the state when you’ve saved for the last time. In the mapping editor, the link Overview at the top of the page will take you back to the datasource configuration overview. After creating a mapping, it will appear in the section <small>Schemas</small>. The entry will show the number of mapped elements (0 for now) and the schema namespace. Pressing the trash can symbol will move it to the trash can, which means that your web service does not support this schema anymore; if you did this by mistake, you can restore it by pressing the <small>Restore</small> button. <small>Purge</small> will delete the schema mapping permanently: You can map several schemas for a given datasource. For example, in addition to the current ABCD2 standard, you could add the deprecated version 1.2 (even though I have no idea why you should do that) or DarwinCore. Just select the appropriate schema in the drop-down list and click <small>Create</small>. Since the namespace of a schema will be used in requests, you cannot create two mappings for the same namespace. Trying this will get you an error message. == Adding Mandatory ABCD Elements == Clicking on a schema in the datasource overview page will take you again to the mapping editor. In the lower half of the page, you will see the basic structure of an ABCD document: Technical/Content Contact: These trees hold information on who should be contacted for technical issues or questions concerning the published database, each with name, address, email address and telephone number. Metadata: In this sub tree you’ll find all metadata fields, for example title and description of the dataset, ownership information, date last modified, IPR statements etc. Units: This part stores the “real” data you want to publish, that means all specimen or observation data. The three elements already shown in the tree are identifiers that make up the so called “Triple ID”, which uniquely identifies records within primary biodiversity networks such as BioCASe and GBIF: - ‘’UnitID’’, which is a unique identifier for each record to be published (you can use the catalogue number, for example, or an existing barcode); - ‘’ SourceInstitutionID’’, an identifier for the institution publishing the dataset (for example BGBM); - ‘’ SourceID’’, an identifier for the dataset (e.g. “Herbarium Berolinense’’). The elements already shown in the mapping editor are printed in red because they are mandatory, which means they are required for an ABCD document. Without them being mapped (to database fields that are non-empty), the Provider Software will not be able to construct valid ABCD documents and will not publish any records. So the first elements that should be mapped are the mandatory fields. To add a mapping, click the <strong>+</strong> symbol next to the field, which will open the mapping editor dialog, where you can select the database table/column you want to map the ABCD element to: In this case, the element /DataSets/DataSet/Metadata/RevisionData/DateModified is mapped to the column ‘’source_update’’ of the ‘’metadata’’ table, which has the data type ‘’date’’. (Remember that table/column retrieval is not supported for all DBMS, so if you’re unlucky, you might need to type in table/column name manually.) <small>Save</small> will accept your settings and close the dialog, <small>Cancel</small> will discard any changes. Instead of mapping an element to a database field, you can map it to a literal. For example, you could enter the technical and content contact information directly as literals in the mapping editor dialog (assuming they’re the same for all records). However, we strongly discourage you from doing this. For one thing, this is just bad style, since it will scatter the information published over different places: Some will be retrieved from the database, some (the literals) will be loaded from the configuration files. Changing the metadata of your service later would then require access to the database and access to the BioCASe configuration – that means the configuration tool password. Be nice to the person who will do these changes and put everything neatly in a database table! The second reason is even more important: Using literals for an element will result in this element to become not searchable, which means it cannot be used in a filter of a Search or a Scan request. Even though this behaviour is discoverable with a Capabilities request (by evaluating the ‘’searchable’’ flag), a potential user of your web service might just not think of that, so it’s a good idea to avoid that in first place. Please use literals sparsely! In the mapping editor window, you can select several source columns and several literals, even a combination of both. The Provider Software will simply concatenate the values retrieved from the database and the literals specified (in the order they’re given). But again: Keep in mind that this will make the element not searchable. A better solution is to do any concatenation operations in a database view. Use the mapping editor window to add mappings for all eight mandatory fields: Click on the <strong>+</strong> symbol next to the element, choose table/column name and the correct data type, then press <small>Save</small>. When you’re done, the page should look similar to the following image. Press <small>Save</small> in the main window to write your changes to the configuration files. It should now look similar to this: == Initial Testing of the Data Source == Once you’ve mapped all mandatory elements, you can do the first test of your BioCASe web service. Even though you haven’t mapped any real information yet, it is a good idea to check the service and solve any problems hitherto before continuing with the mapping. Click on <small>Test mapping!</small> to open the Query Form that can be used for this purpose: With the Query Form you can send a BioCASe request to your web service; it will wait for the reply of the web service and display the response document. On top of the page you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ‘’flora’’ being the name of the datasource. Make sure this is correct for your installation! Into the text box below you can enter the request XML text. Because it would be cumbersome to enter the whole BioCASe request, you can use the links below the box to load templates. For now, click on “ABCD2 Search” to fill the box with a BioCASe Search request for documents in the ABCD2 format (the one we’ve just mapped): As you can see, there is a filter set on the ‘’ScientificName’’ element. Since we haven’t mapped this element yet, this won’t work. So let’s remove the filter (all lines between <filter> and </filter>), leaving the text box like this: Pressing <small>Submit</small> will send the request to the web service and – hopefully – display the response document with the correct ABCD records. For the mapping we’ve just created, it should look similar to the following document (use the – symbol next to the trees TechnicalContact, ContentContact and Metadata to collapse them): If you get an error, please check out the ___debugging tutorial__ on how to debug a web service. You should continue the mapping process only after your web service returns results for the basal mapping we’ve just created. == Adding additional ABCD Elements == Once you’ve successfully managed to map the mandatory ABCD elements (“successfully” means the web service returns ABCD documents), you can go on with the mapping process. This is usually an iterative process: You add a handful of elements (for example, country, country code, locality text and coordinates for the gathering site), then you use the Query Form to see if the information ends up where it is supposed to do. This is the reason the Query Form opens up in a separate tab (or window, of you configured your web browser so): In one tab, you use the mapping editor to add new elements; after pressing Save you go to the Query Form tab and see what the response documents look like. Redo this until you’ve mapped all information you want to publish. In order to map new ABCD concepts, tick the check box labelled <small>Show all concepts</small> and press <small>Refresh</small>. The mapping editor will then display the whole ABCD tree, which looks admittedly a bit intimidating. In order to find the concept you’re looking for, use the search function of your browser. If you want to map the country name of the gathering site, for example, hit <small>Ctrl + F</small> (for Firefox, Microsoft IE and Opera) and type in the name of the ABCD element, namely <small>country</small>. The browser will jump to the Country tree of the Gathering element, where you’ll find the elements for country name (Name), ISO country code (ISO3166Code) and the country name in the country’s language (NameDerived): Continue the sequence mapping new elements/testing until you’ve mapped all information you want to publish. You can have a look at the list of recommend and commonly used ABCD elements here. The full ABCD documentation can be found here. Here you can have a look at a sample response ABCD document for a web service with 77 concepts mapped. Of course your documents will look different, because you will probably use different ABCD elements. But still it’s good to get an idea: asdfads fa4864aa0dfacf99287193fe551861fbd080e8f4 76 75 2011-06-30T10:24:21Z JoergHoletschek 3 /* Creating an ABCD Mapping for a Data Source */ wikitext text/x-wiki This tutorial will explain how to set up a mapping for the ABCD2.06 data schema. If you intend to map another schema (e.g. one of the ABCD extension ABCD-EFG, ABCD-DNA or HISPID), you still should take the time to read it. Large portions of the process will be identical. == Access to Biological Collection Data Schema (ABCD) == ABCD in its current version 2.06 is the data schema typically used in conjunction with BioCASe. It is a highly complex XML data schema with about 1,000 elements that is able to store almost every piece of information that can be found in a natural history specimen collection or an observation database. The complexity is due to the fact that it can be used for a wide range of collections/databases – for living and preserved specimens, observations and culture collections, for zoological, botanical, bacterial and viral collections, marine or terrestrial, for herbaria, botanic and zoological gardens. For each of these special types, ABCD features special sections where information specific for this type can be stored. Other sections of ABCD will be shared by all these types, for example for gathering site, identifier or metadata. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a list of commonly used ABCD elements on the Wiki page [[CommonABCD2Concepts]]. == Creating an ABCD Mapping for a Data Source == If you created a data source from a template (e.g. ''abcdmetadata''), a schema mapping for ABCD will be already existent. In this case, you can directly jump to the section [[#The Mapping Editor]]. If you created an empty data source, the section <small>Schemas</small> in the data source configuration overview will be empty: [[File:abcdSchemaMappings.png]] In order to create a new schema mapping, select the desired schema from the list and click <small>Create</small>. For ABCD, choose <small>ABCD_2.06.xml</small> (be careful not to hit one of the ABCD extensions, they look quite similar). After clicking <small>Create</small>, you will be directed to the mapping editor, which will allow you to edit the mappings. == The Mapping Editor == The mapping editor allows you to add, remove and edit ABCD mappings. On top you’ll find some summary information, on the bottom you’ll see the structure of the ABCD mapping: First thing we will do is to set the root table, which is the table that holds the records you want to publish (specimens or observations). That table must hold one record per object to be published, identified by a unique identifier (called ‘’UnitID’’). This identifier must not be the primary key, as long as it’s unique (if that sentence confuses you, ignore it and just keep in mind that the UnitID must be unique.) Use the drop down box labelled <small>Root table Alias</small> to select the root table of your data model. If your metadata table holds just one record (because all occurrences published share the same metadata) and if it is not linked to the root table by a foreign key, you can specify this table in the drop-down box <small>Static table alias</small>. The Provider Software will use a natural join to connect the metadata record stored in this table to all occurrences published. If your metadata table is referenced by a foreign key in the root able, you must not declare the metadata table as a static table! So either join the metadata table using a foreign key OR declare it as a static table. When you’ve set the root table (and static table, if you use that), press <small>Save</small>. As with the DB structure setup, this will write your changes to the configuration files, closing the tab without saving will discard your changes! Also, in case you messed something up, you can use the Revert button to restore the state when you’ve saved for the last time. In the mapping editor, the link Overview at the top of the page will take you back to the datasource configuration overview. After creating a mapping, it will appear in the section <small>Schemas</small>. The entry will show the number of mapped elements (0 for now) and the schema namespace. Pressing the trash can symbol will move it to the trash can, which means that your web service does not support this schema anymore; if you did this by mistake, you can restore it by pressing the <small>Restore</small> button. <small>Purge</small> will delete the schema mapping permanently: You can map several schemas for a given datasource. For example, in addition to the current ABCD2 standard, you could add the deprecated version 1.2 (even though I have no idea why you should do that) or DarwinCore. Just select the appropriate schema in the drop-down list and click <small>Create</small>. Since the namespace of a schema will be used in requests, you cannot create two mappings for the same namespace. Trying this will get you an error message. == Adding Mandatory ABCD Elements == Clicking on a schema in the datasource overview page will take you again to the mapping editor. In the lower half of the page, you will see the basic structure of an ABCD document: Technical/Content Contact: These trees hold information on who should be contacted for technical issues or questions concerning the published database, each with name, address, email address and telephone number. Metadata: In this sub tree you’ll find all metadata fields, for example title and description of the dataset, ownership information, date last modified, IPR statements etc. Units: This part stores the “real” data you want to publish, that means all specimen or observation data. The three elements already shown in the tree are identifiers that make up the so called “Triple ID”, which uniquely identifies records within primary biodiversity networks such as BioCASe and GBIF: - ‘’UnitID’’, which is a unique identifier for each record to be published (you can use the catalogue number, for example, or an existing barcode); - ‘’ SourceInstitutionID’’, an identifier for the institution publishing the dataset (for example BGBM); - ‘’ SourceID’’, an identifier for the dataset (e.g. “Herbarium Berolinense’’). The elements already shown in the mapping editor are printed in red because they are mandatory, which means they are required for an ABCD document. Without them being mapped (to database fields that are non-empty), the Provider Software will not be able to construct valid ABCD documents and will not publish any records. So the first elements that should be mapped are the mandatory fields. To add a mapping, click the <strong>+</strong> symbol next to the field, which will open the mapping editor dialog, where you can select the database table/column you want to map the ABCD element to: In this case, the element /DataSets/DataSet/Metadata/RevisionData/DateModified is mapped to the column ‘’source_update’’ of the ‘’metadata’’ table, which has the data type ‘’date’’. (Remember that table/column retrieval is not supported for all DBMS, so if you’re unlucky, you might need to type in table/column name manually.) <small>Save</small> will accept your settings and close the dialog, <small>Cancel</small> will discard any changes. Instead of mapping an element to a database field, you can map it to a literal. For example, you could enter the technical and content contact information directly as literals in the mapping editor dialog (assuming they’re the same for all records). However, we strongly discourage you from doing this. For one thing, this is just bad style, since it will scatter the information published over different places: Some will be retrieved from the database, some (the literals) will be loaded from the configuration files. Changing the metadata of your service later would then require access to the database and access to the BioCASe configuration – that means the configuration tool password. Be nice to the person who will do these changes and put everything neatly in a database table! The second reason is even more important: Using literals for an element will result in this element to become not searchable, which means it cannot be used in a filter of a Search or a Scan request. Even though this behaviour is discoverable with a Capabilities request (by evaluating the ‘’searchable’’ flag), a potential user of your web service might just not think of that, so it’s a good idea to avoid that in first place. Please use literals sparsely! In the mapping editor window, you can select several source columns and several literals, even a combination of both. The Provider Software will simply concatenate the values retrieved from the database and the literals specified (in the order they’re given). But again: Keep in mind that this will make the element not searchable. A better solution is to do any concatenation operations in a database view. Use the mapping editor window to add mappings for all eight mandatory fields: Click on the <strong>+</strong> symbol next to the element, choose table/column name and the correct data type, then press <small>Save</small>. When you’re done, the page should look similar to the following image. Press <small>Save</small> in the main window to write your changes to the configuration files. It should now look similar to this: == Initial Testing of the Data Source == Once you’ve mapped all mandatory elements, you can do the first test of your BioCASe web service. Even though you haven’t mapped any real information yet, it is a good idea to check the service and solve any problems hitherto before continuing with the mapping. Click on <small>Test mapping!</small> to open the Query Form that can be used for this purpose: With the Query Form you can send a BioCASe request to your web service; it will wait for the reply of the web service and display the response document. On top of the page you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ‘’flora’’ being the name of the datasource. Make sure this is correct for your installation! Into the text box below you can enter the request XML text. Because it would be cumbersome to enter the whole BioCASe request, you can use the links below the box to load templates. For now, click on “ABCD2 Search” to fill the box with a BioCASe Search request for documents in the ABCD2 format (the one we’ve just mapped): As you can see, there is a filter set on the ‘’ScientificName’’ element. Since we haven’t mapped this element yet, this won’t work. So let’s remove the filter (all lines between <filter> and </filter>), leaving the text box like this: Pressing <small>Submit</small> will send the request to the web service and – hopefully – display the response document with the correct ABCD records. For the mapping we’ve just created, it should look similar to the following document (use the – symbol next to the trees TechnicalContact, ContentContact and Metadata to collapse them): If you get an error, please check out the ___debugging tutorial__ on how to debug a web service. You should continue the mapping process only after your web service returns results for the basal mapping we’ve just created. == Adding additional ABCD Elements == Once you’ve successfully managed to map the mandatory ABCD elements (“successfully” means the web service returns ABCD documents), you can go on with the mapping process. This is usually an iterative process: You add a handful of elements (for example, country, country code, locality text and coordinates for the gathering site), then you use the Query Form to see if the information ends up where it is supposed to do. This is the reason the Query Form opens up in a separate tab (or window, of you configured your web browser so): In one tab, you use the mapping editor to add new elements; after pressing Save you go to the Query Form tab and see what the response documents look like. Redo this until you’ve mapped all information you want to publish. In order to map new ABCD concepts, tick the check box labelled <small>Show all concepts</small> and press <small>Refresh</small>. The mapping editor will then display the whole ABCD tree, which looks admittedly a bit intimidating. In order to find the concept you’re looking for, use the search function of your browser. If you want to map the country name of the gathering site, for example, hit <small>Ctrl + F</small> (for Firefox, Microsoft IE and Opera) and type in the name of the ABCD element, namely <small>country</small>. The browser will jump to the Country tree of the Gathering element, where you’ll find the elements for country name (Name), ISO country code (ISO3166Code) and the country name in the country’s language (NameDerived): Continue the sequence mapping new elements/testing until you’ve mapped all information you want to publish. You can have a look at the list of recommend and commonly used ABCD elements here. The full ABCD documentation can be found here. Here you can have a look at a sample response ABCD document for a web service with 77 concepts mapped. Of course your documents will look different, because you will probably use different ABCD elements. But still it’s good to get an idea: asdfads 41c13f792742b68492ef52535d2da5b90a95c592 77 76 2011-06-30T10:26:49Z JoergHoletschek 3 /* Creating an ABCD Mapping for a Data Source */ wikitext text/x-wiki This tutorial will explain how to set up a mapping for the ABCD2.06 data schema. If you intend to map another schema (e.g. one of the ABCD extension ABCD-EFG, ABCD-DNA or HISPID), you still should take the time to read it. Large portions of the process will be identical. == Access to Biological Collection Data Schema (ABCD) == ABCD in its current version 2.06 is the data schema typically used in conjunction with BioCASe. It is a highly complex XML data schema with about 1,000 elements that is able to store almost every piece of information that can be found in a natural history specimen collection or an observation database. The complexity is due to the fact that it can be used for a wide range of collections/databases – for living and preserved specimens, observations and culture collections, for zoological, botanical, bacterial and viral collections, marine or terrestrial, for herbaria, botanic and zoological gardens. For each of these special types, ABCD features special sections where information specific for this type can be stored. Other sections of ABCD will be shared by all these types, for example for gathering site, identifier or metadata. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a list of commonly used ABCD elements on the Wiki page [[CommonABCD2Concepts]]. == Creating an ABCD Mapping for a Data Source == If you created a data source from a template (e.g. ''abcdmetadata''), a schema mapping for ABCD will be already existent. In this case, you can directly jump to the next section. If you created an empty data source, the section <small>Schemas</small> in the data source configuration overview will be empty: [[File:abcdSchemaMappings.png]] In order to create a new schema mapping, select the desired schema from the list and click <small>Create</small>. For ABCD, choose <small>ABCD_2.06.xml</small> (be careful not to hit one of the ABCD extensions, they look quite similar). After clicking <small>Create</small>, you will be directed to the mapping editor, which will allow you to edit the mappings. == The Mapping Editor == The mapping editor allows you to add, remove and edit ABCD mappings. On top you’ll find some summary information, on the bottom you’ll see the structure of the ABCD mapping: First thing we will do is to set the root table, which is the table that holds the records you want to publish (specimens or observations). That table must hold one record per object to be published, identified by a unique identifier (called ‘’UnitID’’). This identifier must not be the primary key, as long as it’s unique (if that sentence confuses you, ignore it and just keep in mind that the UnitID must be unique.) Use the drop down box labelled <small>Root table Alias</small> to select the root table of your data model. If your metadata table holds just one record (because all occurrences published share the same metadata) and if it is not linked to the root table by a foreign key, you can specify this table in the drop-down box <small>Static table alias</small>. The Provider Software will use a natural join to connect the metadata record stored in this table to all occurrences published. If your metadata table is referenced by a foreign key in the root able, you must not declare the metadata table as a static table! So either join the metadata table using a foreign key OR declare it as a static table. When you’ve set the root table (and static table, if you use that), press <small>Save</small>. As with the DB structure setup, this will write your changes to the configuration files, closing the tab without saving will discard your changes! Also, in case you messed something up, you can use the Revert button to restore the state when you’ve saved for the last time. In the mapping editor, the link Overview at the top of the page will take you back to the datasource configuration overview. After creating a mapping, it will appear in the section <small>Schemas</small>. The entry will show the number of mapped elements (0 for now) and the schema namespace. Pressing the trash can symbol will move it to the trash can, which means that your web service does not support this schema anymore; if you did this by mistake, you can restore it by pressing the <small>Restore</small> button. <small>Purge</small> will delete the schema mapping permanently: You can map several schemas for a given datasource. For example, in addition to the current ABCD2 standard, you could add the deprecated version 1.2 (even though I have no idea why you should do that) or DarwinCore. Just select the appropriate schema in the drop-down list and click <small>Create</small>. Since the namespace of a schema will be used in requests, you cannot create two mappings for the same namespace. Trying this will get you an error message. == Adding Mandatory ABCD Elements == Clicking on a schema in the datasource overview page will take you again to the mapping editor. In the lower half of the page, you will see the basic structure of an ABCD document: Technical/Content Contact: These trees hold information on who should be contacted for technical issues or questions concerning the published database, each with name, address, email address and telephone number. Metadata: In this sub tree you’ll find all metadata fields, for example title and description of the dataset, ownership information, date last modified, IPR statements etc. Units: This part stores the “real” data you want to publish, that means all specimen or observation data. The three elements already shown in the tree are identifiers that make up the so called “Triple ID”, which uniquely identifies records within primary biodiversity networks such as BioCASe and GBIF: - ‘’UnitID’’, which is a unique identifier for each record to be published (you can use the catalogue number, for example, or an existing barcode); - ‘’ SourceInstitutionID’’, an identifier for the institution publishing the dataset (for example BGBM); - ‘’ SourceID’’, an identifier for the dataset (e.g. “Herbarium Berolinense’’). The elements already shown in the mapping editor are printed in red because they are mandatory, which means they are required for an ABCD document. Without them being mapped (to database fields that are non-empty), the Provider Software will not be able to construct valid ABCD documents and will not publish any records. So the first elements that should be mapped are the mandatory fields. To add a mapping, click the <strong>+</strong> symbol next to the field, which will open the mapping editor dialog, where you can select the database table/column you want to map the ABCD element to: In this case, the element /DataSets/DataSet/Metadata/RevisionData/DateModified is mapped to the column ‘’source_update’’ of the ‘’metadata’’ table, which has the data type ‘’date’’. (Remember that table/column retrieval is not supported for all DBMS, so if you’re unlucky, you might need to type in table/column name manually.) <small>Save</small> will accept your settings and close the dialog, <small>Cancel</small> will discard any changes. Instead of mapping an element to a database field, you can map it to a literal. For example, you could enter the technical and content contact information directly as literals in the mapping editor dialog (assuming they’re the same for all records). However, we strongly discourage you from doing this. For one thing, this is just bad style, since it will scatter the information published over different places: Some will be retrieved from the database, some (the literals) will be loaded from the configuration files. Changing the metadata of your service later would then require access to the database and access to the BioCASe configuration – that means the configuration tool password. Be nice to the person who will do these changes and put everything neatly in a database table! The second reason is even more important: Using literals for an element will result in this element to become not searchable, which means it cannot be used in a filter of a Search or a Scan request. Even though this behaviour is discoverable with a Capabilities request (by evaluating the ‘’searchable’’ flag), a potential user of your web service might just not think of that, so it’s a good idea to avoid that in first place. Please use literals sparsely! In the mapping editor window, you can select several source columns and several literals, even a combination of both. The Provider Software will simply concatenate the values retrieved from the database and the literals specified (in the order they’re given). But again: Keep in mind that this will make the element not searchable. A better solution is to do any concatenation operations in a database view. Use the mapping editor window to add mappings for all eight mandatory fields: Click on the <strong>+</strong> symbol next to the element, choose table/column name and the correct data type, then press <small>Save</small>. When you’re done, the page should look similar to the following image. Press <small>Save</small> in the main window to write your changes to the configuration files. It should now look similar to this: == Initial Testing of the Data Source == Once you’ve mapped all mandatory elements, you can do the first test of your BioCASe web service. Even though you haven’t mapped any real information yet, it is a good idea to check the service and solve any problems hitherto before continuing with the mapping. Click on <small>Test mapping!</small> to open the Query Form that can be used for this purpose: With the Query Form you can send a BioCASe request to your web service; it will wait for the reply of the web service and display the response document. On top of the page you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ‘’flora’’ being the name of the datasource. Make sure this is correct for your installation! Into the text box below you can enter the request XML text. Because it would be cumbersome to enter the whole BioCASe request, you can use the links below the box to load templates. For now, click on “ABCD2 Search” to fill the box with a BioCASe Search request for documents in the ABCD2 format (the one we’ve just mapped): As you can see, there is a filter set on the ‘’ScientificName’’ element. Since we haven’t mapped this element yet, this won’t work. So let’s remove the filter (all lines between <filter> and </filter>), leaving the text box like this: Pressing <small>Submit</small> will send the request to the web service and – hopefully – display the response document with the correct ABCD records. For the mapping we’ve just created, it should look similar to the following document (use the – symbol next to the trees TechnicalContact, ContentContact and Metadata to collapse them): If you get an error, please check out the ___debugging tutorial__ on how to debug a web service. You should continue the mapping process only after your web service returns results for the basal mapping we’ve just created. == Adding additional ABCD Elements == Once you’ve successfully managed to map the mandatory ABCD elements (“successfully” means the web service returns ABCD documents), you can go on with the mapping process. This is usually an iterative process: You add a handful of elements (for example, country, country code, locality text and coordinates for the gathering site), then you use the Query Form to see if the information ends up where it is supposed to do. This is the reason the Query Form opens up in a separate tab (or window, of you configured your web browser so): In one tab, you use the mapping editor to add new elements; after pressing Save you go to the Query Form tab and see what the response documents look like. Redo this until you’ve mapped all information you want to publish. In order to map new ABCD concepts, tick the check box labelled <small>Show all concepts</small> and press <small>Refresh</small>. The mapping editor will then display the whole ABCD tree, which looks admittedly a bit intimidating. In order to find the concept you’re looking for, use the search function of your browser. If you want to map the country name of the gathering site, for example, hit <small>Ctrl + F</small> (for Firefox, Microsoft IE and Opera) and type in the name of the ABCD element, namely <small>country</small>. The browser will jump to the Country tree of the Gathering element, where you’ll find the elements for country name (Name), ISO country code (ISO3166Code) and the country name in the country’s language (NameDerived): Continue the sequence mapping new elements/testing until you’ve mapped all information you want to publish. You can have a look at the list of recommend and commonly used ABCD elements here. The full ABCD documentation can be found here. Here you can have a look at a sample response ABCD document for a web service with 77 concepts mapped. Of course your documents will look different, because you will probably use different ABCD elements. But still it’s good to get an idea: asdfads ac6ec5b9ca17df0a42a8bc9163c0e00ab780bc24 79 77 2011-06-30T10:30:02Z JoergHoletschek 3 /* The Mapping Editor */ wikitext text/x-wiki This tutorial will explain how to set up a mapping for the ABCD2.06 data schema. If you intend to map another schema (e.g. one of the ABCD extension ABCD-EFG, ABCD-DNA or HISPID), you still should take the time to read it. Large portions of the process will be identical. == Access to Biological Collection Data Schema (ABCD) == ABCD in its current version 2.06 is the data schema typically used in conjunction with BioCASe. It is a highly complex XML data schema with about 1,000 elements that is able to store almost every piece of information that can be found in a natural history specimen collection or an observation database. The complexity is due to the fact that it can be used for a wide range of collections/databases – for living and preserved specimens, observations and culture collections, for zoological, botanical, bacterial and viral collections, marine or terrestrial, for herbaria, botanic and zoological gardens. For each of these special types, ABCD features special sections where information specific for this type can be stored. Other sections of ABCD will be shared by all these types, for example for gathering site, identifier or metadata. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a list of commonly used ABCD elements on the Wiki page [[CommonABCD2Concepts]]. == Creating an ABCD Mapping for a Data Source == If you created a data source from a template (e.g. ''abcdmetadata''), a schema mapping for ABCD will be already existent. In this case, you can directly jump to the next section. If you created an empty data source, the section <small>Schemas</small> in the data source configuration overview will be empty: [[File:abcdSchemaMappings.png]] In order to create a new schema mapping, select the desired schema from the list and click <small>Create</small>. For ABCD, choose <small>ABCD_2.06.xml</small> (be careful not to hit one of the ABCD extensions, they look quite similar). After clicking <small>Create</small>, you will be directed to the mapping editor, which will allow you to edit the mappings. == The Mapping Editor == The mapping editor allows you to add, remove and edit ABCD mappings. On top you’ll find some summary information, on the bottom you’ll see the structure of the ABCD mapping: [[File:abcdMappingEditorEmpty.png]] First thing we will do is to set the root table, which is the table that holds the records you want to publish (specimens or observations). That table must hold one record per object to be published, identified by a unique identifier (called ''UnitID''). This identifier must not be the primary key, as long as it is unique (if that sentence confuses you, ignore it and just keep in mind that the UnitID must be unique.) Use the drop down box labelled <small>Root table Alias</small> to select the root table of your data model. If your metadata table holds just one record (because all occurrences published share the same metadata) and if it is not linked to the root table by a foreign key, you can specify this table in the drop-down box <small>Static table Alias</small>. The Provider Software will use a natural join to connect the metadata record stored in this table to all occurrences published. If your metadata table is referenced by a foreign key in the root able, you must not declare the metadata table as a static table! So either join the metadata table using a foreign key OR declare it as a static table. When you’ve set the root table (and static table, if you use that), press <small>Save</small>. As with the DB structure setup, this will write your changes to the configuration files, closing the tab without saving will discard your changes! Also, in case you messed something up, you can use the <small>Revert</small> button to restore the state when you’ve saved for the last time. In the mapping editor, the link <small>Overview</small> at the top of the page will take you back to the datasource configuration overview. After creating a mapping, it will appear in the section <small>Schemas</small>. The entry will show the number of mapped elements (0 for now) and the schema namespace. Pressing the trash can symbol will move it to the trash can, which means that your web service does not support this schema anymore; if you did this by mistake, you can restore it by pressing the <small>Restore</small> button. <small>Purge</small> will delete the schema mapping permanently: [[File:abcdSchemaMappings2.png]] You can map several schemas for a given datasource. For example, in addition to the current ABCD2 standard, you could add the deprecated version 1.2 (even though I have no idea why you should do that) or DarwinCore. Just select the appropriate schema in the drop-down list and click <small>Create</small>. Since the namespace of a schema will be used in requests, you cannot create two mappings for the same namespace. Trying this will get you an error message. == Adding Mandatory ABCD Elements == Clicking on a schema in the datasource overview page will take you again to the mapping editor. In the lower half of the page, you will see the basic structure of an ABCD document: Technical/Content Contact: These trees hold information on who should be contacted for technical issues or questions concerning the published database, each with name, address, email address and telephone number. Metadata: In this sub tree you’ll find all metadata fields, for example title and description of the dataset, ownership information, date last modified, IPR statements etc. Units: This part stores the “real” data you want to publish, that means all specimen or observation data. The three elements already shown in the tree are identifiers that make up the so called “Triple ID”, which uniquely identifies records within primary biodiversity networks such as BioCASe and GBIF: - ‘’UnitID’’, which is a unique identifier for each record to be published (you can use the catalogue number, for example, or an existing barcode); - ‘’ SourceInstitutionID’’, an identifier for the institution publishing the dataset (for example BGBM); - ‘’ SourceID’’, an identifier for the dataset (e.g. “Herbarium Berolinense’’). The elements already shown in the mapping editor are printed in red because they are mandatory, which means they are required for an ABCD document. Without them being mapped (to database fields that are non-empty), the Provider Software will not be able to construct valid ABCD documents and will not publish any records. So the first elements that should be mapped are the mandatory fields. To add a mapping, click the <strong>+</strong> symbol next to the field, which will open the mapping editor dialog, where you can select the database table/column you want to map the ABCD element to: In this case, the element /DataSets/DataSet/Metadata/RevisionData/DateModified is mapped to the column ‘’source_update’’ of the ‘’metadata’’ table, which has the data type ‘’date’’. (Remember that table/column retrieval is not supported for all DBMS, so if you’re unlucky, you might need to type in table/column name manually.) <small>Save</small> will accept your settings and close the dialog, <small>Cancel</small> will discard any changes. Instead of mapping an element to a database field, you can map it to a literal. For example, you could enter the technical and content contact information directly as literals in the mapping editor dialog (assuming they’re the same for all records). However, we strongly discourage you from doing this. For one thing, this is just bad style, since it will scatter the information published over different places: Some will be retrieved from the database, some (the literals) will be loaded from the configuration files. Changing the metadata of your service later would then require access to the database and access to the BioCASe configuration – that means the configuration tool password. Be nice to the person who will do these changes and put everything neatly in a database table! The second reason is even more important: Using literals for an element will result in this element to become not searchable, which means it cannot be used in a filter of a Search or a Scan request. Even though this behaviour is discoverable with a Capabilities request (by evaluating the ‘’searchable’’ flag), a potential user of your web service might just not think of that, so it’s a good idea to avoid that in first place. Please use literals sparsely! In the mapping editor window, you can select several source columns and several literals, even a combination of both. The Provider Software will simply concatenate the values retrieved from the database and the literals specified (in the order they’re given). But again: Keep in mind that this will make the element not searchable. A better solution is to do any concatenation operations in a database view. Use the mapping editor window to add mappings for all eight mandatory fields: Click on the <strong>+</strong> symbol next to the element, choose table/column name and the correct data type, then press <small>Save</small>. When you’re done, the page should look similar to the following image. Press <small>Save</small> in the main window to write your changes to the configuration files. It should now look similar to this: == Initial Testing of the Data Source == Once you’ve mapped all mandatory elements, you can do the first test of your BioCASe web service. Even though you haven’t mapped any real information yet, it is a good idea to check the service and solve any problems hitherto before continuing with the mapping. Click on <small>Test mapping!</small> to open the Query Form that can be used for this purpose: With the Query Form you can send a BioCASe request to your web service; it will wait for the reply of the web service and display the response document. On top of the page you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ‘’flora’’ being the name of the datasource. Make sure this is correct for your installation! Into the text box below you can enter the request XML text. Because it would be cumbersome to enter the whole BioCASe request, you can use the links below the box to load templates. For now, click on “ABCD2 Search” to fill the box with a BioCASe Search request for documents in the ABCD2 format (the one we’ve just mapped): As you can see, there is a filter set on the ‘’ScientificName’’ element. Since we haven’t mapped this element yet, this won’t work. So let’s remove the filter (all lines between <filter> and </filter>), leaving the text box like this: Pressing <small>Submit</small> will send the request to the web service and – hopefully – display the response document with the correct ABCD records. For the mapping we’ve just created, it should look similar to the following document (use the – symbol next to the trees TechnicalContact, ContentContact and Metadata to collapse them): If you get an error, please check out the ___debugging tutorial__ on how to debug a web service. You should continue the mapping process only after your web service returns results for the basal mapping we’ve just created. == Adding additional ABCD Elements == Once you’ve successfully managed to map the mandatory ABCD elements (“successfully” means the web service returns ABCD documents), you can go on with the mapping process. This is usually an iterative process: You add a handful of elements (for example, country, country code, locality text and coordinates for the gathering site), then you use the Query Form to see if the information ends up where it is supposed to do. This is the reason the Query Form opens up in a separate tab (or window, of you configured your web browser so): In one tab, you use the mapping editor to add new elements; after pressing Save you go to the Query Form tab and see what the response documents look like. Redo this until you’ve mapped all information you want to publish. In order to map new ABCD concepts, tick the check box labelled <small>Show all concepts</small> and press <small>Refresh</small>. The mapping editor will then display the whole ABCD tree, which looks admittedly a bit intimidating. In order to find the concept you’re looking for, use the search function of your browser. If you want to map the country name of the gathering site, for example, hit <small>Ctrl + F</small> (for Firefox, Microsoft IE and Opera) and type in the name of the ABCD element, namely <small>country</small>. The browser will jump to the Country tree of the Gathering element, where you’ll find the elements for country name (Name), ISO country code (ISO3166Code) and the country name in the country’s language (NameDerived): Continue the sequence mapping new elements/testing until you’ve mapped all information you want to publish. You can have a look at the list of recommend and commonly used ABCD elements here. The full ABCD documentation can be found here. Here you can have a look at a sample response ABCD document for a web service with 77 concepts mapped. Of course your documents will look different, because you will probably use different ABCD elements. But still it’s good to get an idea: asdfads 5d47bbbd5765cd7b30c6d72c0bf83fccb87ae26d 82 79 2011-06-30T10:36:55Z JoergHoletschek 3 /* Adding Mandatory ABCD Elements */ wikitext text/x-wiki This tutorial will explain how to set up a mapping for the ABCD2.06 data schema. If you intend to map another schema (e.g. one of the ABCD extension ABCD-EFG, ABCD-DNA or HISPID), you still should take the time to read it. Large portions of the process will be identical. == Access to Biological Collection Data Schema (ABCD) == ABCD in its current version 2.06 is the data schema typically used in conjunction with BioCASe. It is a highly complex XML data schema with about 1,000 elements that is able to store almost every piece of information that can be found in a natural history specimen collection or an observation database. The complexity is due to the fact that it can be used for a wide range of collections/databases – for living and preserved specimens, observations and culture collections, for zoological, botanical, bacterial and viral collections, marine or terrestrial, for herbaria, botanic and zoological gardens. For each of these special types, ABCD features special sections where information specific for this type can be stored. Other sections of ABCD will be shared by all these types, for example for gathering site, identifier or metadata. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a list of commonly used ABCD elements on the Wiki page [[CommonABCD2Concepts]]. == Creating an ABCD Mapping for a Data Source == If you created a data source from a template (e.g. ''abcdmetadata''), a schema mapping for ABCD will be already existent. In this case, you can directly jump to the next section. If you created an empty data source, the section <small>Schemas</small> in the data source configuration overview will be empty: [[File:abcdSchemaMappings.png]] In order to create a new schema mapping, select the desired schema from the list and click <small>Create</small>. For ABCD, choose <small>ABCD_2.06.xml</small> (be careful not to hit one of the ABCD extensions, they look quite similar). After clicking <small>Create</small>, you will be directed to the mapping editor, which will allow you to edit the mappings. == The Mapping Editor == The mapping editor allows you to add, remove and edit ABCD mappings. On top you’ll find some summary information, on the bottom you’ll see the structure of the ABCD mapping: [[File:abcdMappingEditorEmpty.png]] First thing we will do is to set the root table, which is the table that holds the records you want to publish (specimens or observations). That table must hold one record per object to be published, identified by a unique identifier (called ''UnitID''). This identifier must not be the primary key, as long as it is unique (if that sentence confuses you, ignore it and just keep in mind that the UnitID must be unique.) Use the drop down box labelled <small>Root table Alias</small> to select the root table of your data model. If your metadata table holds just one record (because all occurrences published share the same metadata) and if it is not linked to the root table by a foreign key, you can specify this table in the drop-down box <small>Static table Alias</small>. The Provider Software will use a natural join to connect the metadata record stored in this table to all occurrences published. If your metadata table is referenced by a foreign key in the root able, you must not declare the metadata table as a static table! So either join the metadata table using a foreign key OR declare it as a static table. When you’ve set the root table (and static table, if you use that), press <small>Save</small>. As with the DB structure setup, this will write your changes to the configuration files, closing the tab without saving will discard your changes! Also, in case you messed something up, you can use the <small>Revert</small> button to restore the state when you’ve saved for the last time. In the mapping editor, the link <small>Overview</small> at the top of the page will take you back to the datasource configuration overview. After creating a mapping, it will appear in the section <small>Schemas</small>. The entry will show the number of mapped elements (0 for now) and the schema namespace. Pressing the trash can symbol will move it to the trash can, which means that your web service does not support this schema anymore; if you did this by mistake, you can restore it by pressing the <small>Restore</small> button. <small>Purge</small> will delete the schema mapping permanently: [[File:abcdSchemaMappings2.png]] You can map several schemas for a given datasource. For example, in addition to the current ABCD2 standard, you could add the deprecated version 1.2 (even though I have no idea why you should do that) or DarwinCore. Just select the appropriate schema in the drop-down list and click <small>Create</small>. Since the namespace of a schema will be used in requests, you cannot create two mappings for the same namespace. Trying this will get you an error message. == Adding Mandatory ABCD Elements == Clicking on a schema in the datasource overview page will take you again to the mapping editor. In the lower half of the page, you will see the basic structure of an ABCD document: [[File:abcdMappingEditorEmptyBottom.png]] <strong>Technical/Content Contact:</strong> These trees hold information on who should be contacted for technical issues or questions concerning the published database, each with name, address, email address and telephone number. <strong>Metadata:</strong> In this sub tree you’ll find all metadata fields, for example title and description of the dataset, ownership information, date last modified, IPR statements etc. <strong>Units:</strong> This part stores the "real" data you want to publish, that means all specimen or observation data. The three elements already shown in the tree are identifiers that make up the so called "Triple ID", which uniquely identifies records within primary biodiversity networks such as BioCASe and GBIF: * ''UnitID'', which is a unique identifier for each record to be published (you can use the catalogue number, for example, or an existing barcode); * ''SourceInstitutionID'', an identifier for the institution publishing the dataset (for example BGBM); * ''SourceID'', an identifier for the dataset (e.g. “Herbarium Berolinense’’). The elements already shown in the mapping editor are printed in red because they are mandatory, which means they are required for an ABCD document. Without them being mapped (to database fields that are non-empty), the Provider Software will not be able to construct valid ABCD documents and will not publish any records. So the first elements that should be mapped are the mandatory fields. To add a mapping, click the <strong>+</strong> symbol next to the field, which will open the mapping editor dialog, where you can select the database table/column you want to map the ABCD element to: [[File:abcdMappingEditorDialog.png]] In this case, the element ''/DataSets/DataSet/Metadata/RevisionData/DateModified'' is mapped to the column ''source_update'' of the ''metadata'' table, which has the data type ''date''. (Remember that table/column retrieval is not supported for all DBMS, so if you’re unlucky, you might need to type in table/column name manually.) <small>Save</small> will accept your settings and close the dialog, <small>Cancel</small> will discard any changes. Instead of mapping an element to a database field, you can map it to a literal. For example, you could enter the technical and content contact information directly as literals in the mapping editor dialog (assuming they’re the same for all records). However, we strongly discourage you from doing this. For one thing, this is just bad style, since it will scatter the information published over different places: Some will be retrieved from the database, some (the literals) will be loaded from the configuration files. Changing the metadata of your service later would then require access to the database and access to the BioCASe configuration – that means the configuration tool password. Be nice to the person who will do these changes and put everything neatly in a database table! The second reason is even more important: Using literals for an element will result in this element to become not searchable, which means it cannot be used in a filter of a Search or a Scan request. Even though this behaviour is discoverable with a Capabilities request (by evaluating the ''searchable'' flag), a potential user of your web service might just not think of that, so it’s a good idea to avoid that in first place. Please use literals sparsely! In the mapping editor window, you can select several source columns and several literals, even a combination of both. The Provider Software will simply concatenate the values retrieved from the database and the literals specified (in the order they’re given). But again: Keep in mind that this will make the element not searchable. A better solution is to do any concatenation operations in a database view. Use the mapping editor window to add mappings for all eight mandatory fields: Click on the <strong>+</strong> symbol next to the element, choose table/column name and the correct data type, then press <small>Save</small>. When you’re done, the page should look similar to the following image. Press <small>Save</small> in the main window to write your changes to the configuration files. [[File:abcdMappingEditorMandatoryElements.png]] == Initial Testing of the Data Source == Once you’ve mapped all mandatory elements, you can do the first test of your BioCASe web service. Even though you haven’t mapped any real information yet, it is a good idea to check the service and solve any problems hitherto before continuing with the mapping. Click on <small>Test mapping!</small> to open the Query Form that can be used for this purpose: With the Query Form you can send a BioCASe request to your web service; it will wait for the reply of the web service and display the response document. On top of the page you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ‘’flora’’ being the name of the datasource. Make sure this is correct for your installation! Into the text box below you can enter the request XML text. Because it would be cumbersome to enter the whole BioCASe request, you can use the links below the box to load templates. For now, click on “ABCD2 Search” to fill the box with a BioCASe Search request for documents in the ABCD2 format (the one we’ve just mapped): As you can see, there is a filter set on the ‘’ScientificName’’ element. Since we haven’t mapped this element yet, this won’t work. So let’s remove the filter (all lines between <filter> and </filter>), leaving the text box like this: Pressing <small>Submit</small> will send the request to the web service and – hopefully – display the response document with the correct ABCD records. For the mapping we’ve just created, it should look similar to the following document (use the – symbol next to the trees TechnicalContact, ContentContact and Metadata to collapse them): If you get an error, please check out the ___debugging tutorial__ on how to debug a web service. You should continue the mapping process only after your web service returns results for the basal mapping we’ve just created. == Adding additional ABCD Elements == Once you’ve successfully managed to map the mandatory ABCD elements (“successfully” means the web service returns ABCD documents), you can go on with the mapping process. This is usually an iterative process: You add a handful of elements (for example, country, country code, locality text and coordinates for the gathering site), then you use the Query Form to see if the information ends up where it is supposed to do. This is the reason the Query Form opens up in a separate tab (or window, of you configured your web browser so): In one tab, you use the mapping editor to add new elements; after pressing Save you go to the Query Form tab and see what the response documents look like. Redo this until you’ve mapped all information you want to publish. In order to map new ABCD concepts, tick the check box labelled <small>Show all concepts</small> and press <small>Refresh</small>. The mapping editor will then display the whole ABCD tree, which looks admittedly a bit intimidating. In order to find the concept you’re looking for, use the search function of your browser. If you want to map the country name of the gathering site, for example, hit <small>Ctrl + F</small> (for Firefox, Microsoft IE and Opera) and type in the name of the ABCD element, namely <small>country</small>. The browser will jump to the Country tree of the Gathering element, where you’ll find the elements for country name (Name), ISO country code (ISO3166Code) and the country name in the country’s language (NameDerived): Continue the sequence mapping new elements/testing until you’ve mapped all information you want to publish. You can have a look at the list of recommend and commonly used ABCD elements here. The full ABCD documentation can be found here. Here you can have a look at a sample response ABCD document for a web service with 77 concepts mapped. Of course your documents will look different, because you will probably use different ABCD elements. But still it’s good to get an idea: asdfads 5e3e9e48d064dbd208dc018efb7d74329e42c07c 86 82 2011-06-30T10:44:14Z JoergHoletschek 3 /* Initial Testing of the Data Source */ wikitext text/x-wiki This tutorial will explain how to set up a mapping for the ABCD2.06 data schema. If you intend to map another schema (e.g. one of the ABCD extension ABCD-EFG, ABCD-DNA or HISPID), you still should take the time to read it. Large portions of the process will be identical. == Access to Biological Collection Data Schema (ABCD) == ABCD in its current version 2.06 is the data schema typically used in conjunction with BioCASe. It is a highly complex XML data schema with about 1,000 elements that is able to store almost every piece of information that can be found in a natural history specimen collection or an observation database. The complexity is due to the fact that it can be used for a wide range of collections/databases – for living and preserved specimens, observations and culture collections, for zoological, botanical, bacterial and viral collections, marine or terrestrial, for herbaria, botanic and zoological gardens. For each of these special types, ABCD features special sections where information specific for this type can be stored. Other sections of ABCD will be shared by all these types, for example for gathering site, identifier or metadata. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a list of commonly used ABCD elements on the Wiki page [[CommonABCD2Concepts]]. == Creating an ABCD Mapping for a Data Source == If you created a data source from a template (e.g. ''abcdmetadata''), a schema mapping for ABCD will be already existent. In this case, you can directly jump to the next section. If you created an empty data source, the section <small>Schemas</small> in the data source configuration overview will be empty: [[File:abcdSchemaMappings.png]] In order to create a new schema mapping, select the desired schema from the list and click <small>Create</small>. For ABCD, choose <small>ABCD_2.06.xml</small> (be careful not to hit one of the ABCD extensions, they look quite similar). After clicking <small>Create</small>, you will be directed to the mapping editor, which will allow you to edit the mappings. == The Mapping Editor == The mapping editor allows you to add, remove and edit ABCD mappings. On top you’ll find some summary information, on the bottom you’ll see the structure of the ABCD mapping: [[File:abcdMappingEditorEmpty.png]] First thing we will do is to set the root table, which is the table that holds the records you want to publish (specimens or observations). That table must hold one record per object to be published, identified by a unique identifier (called ''UnitID''). This identifier must not be the primary key, as long as it is unique (if that sentence confuses you, ignore it and just keep in mind that the UnitID must be unique.) Use the drop down box labelled <small>Root table Alias</small> to select the root table of your data model. If your metadata table holds just one record (because all occurrences published share the same metadata) and if it is not linked to the root table by a foreign key, you can specify this table in the drop-down box <small>Static table Alias</small>. The Provider Software will use a natural join to connect the metadata record stored in this table to all occurrences published. If your metadata table is referenced by a foreign key in the root able, you must not declare the metadata table as a static table! So either join the metadata table using a foreign key OR declare it as a static table. When you’ve set the root table (and static table, if you use that), press <small>Save</small>. As with the DB structure setup, this will write your changes to the configuration files, closing the tab without saving will discard your changes! Also, in case you messed something up, you can use the <small>Revert</small> button to restore the state when you’ve saved for the last time. In the mapping editor, the link <small>Overview</small> at the top of the page will take you back to the datasource configuration overview. After creating a mapping, it will appear in the section <small>Schemas</small>. The entry will show the number of mapped elements (0 for now) and the schema namespace. Pressing the trash can symbol will move it to the trash can, which means that your web service does not support this schema anymore; if you did this by mistake, you can restore it by pressing the <small>Restore</small> button. <small>Purge</small> will delete the schema mapping permanently: [[File:abcdSchemaMappings2.png]] You can map several schemas for a given datasource. For example, in addition to the current ABCD2 standard, you could add the deprecated version 1.2 (even though I have no idea why you should do that) or DarwinCore. Just select the appropriate schema in the drop-down list and click <small>Create</small>. Since the namespace of a schema will be used in requests, you cannot create two mappings for the same namespace. Trying this will get you an error message. == Adding Mandatory ABCD Elements == Clicking on a schema in the datasource overview page will take you again to the mapping editor. In the lower half of the page, you will see the basic structure of an ABCD document: [[File:abcdMappingEditorEmptyBottom.png]] <strong>Technical/Content Contact:</strong> These trees hold information on who should be contacted for technical issues or questions concerning the published database, each with name, address, email address and telephone number. <strong>Metadata:</strong> In this sub tree you’ll find all metadata fields, for example title and description of the dataset, ownership information, date last modified, IPR statements etc. <strong>Units:</strong> This part stores the "real" data you want to publish, that means all specimen or observation data. The three elements already shown in the tree are identifiers that make up the so called "Triple ID", which uniquely identifies records within primary biodiversity networks such as BioCASe and GBIF: * ''UnitID'', which is a unique identifier for each record to be published (you can use the catalogue number, for example, or an existing barcode); * ''SourceInstitutionID'', an identifier for the institution publishing the dataset (for example BGBM); * ''SourceID'', an identifier for the dataset (e.g. “Herbarium Berolinense’’). The elements already shown in the mapping editor are printed in red because they are mandatory, which means they are required for an ABCD document. Without them being mapped (to database fields that are non-empty), the Provider Software will not be able to construct valid ABCD documents and will not publish any records. So the first elements that should be mapped are the mandatory fields. To add a mapping, click the <strong>+</strong> symbol next to the field, which will open the mapping editor dialog, where you can select the database table/column you want to map the ABCD element to: [[File:abcdMappingEditorDialog.png]] In this case, the element ''/DataSets/DataSet/Metadata/RevisionData/DateModified'' is mapped to the column ''source_update'' of the ''metadata'' table, which has the data type ''date''. (Remember that table/column retrieval is not supported for all DBMS, so if you’re unlucky, you might need to type in table/column name manually.) <small>Save</small> will accept your settings and close the dialog, <small>Cancel</small> will discard any changes. Instead of mapping an element to a database field, you can map it to a literal. For example, you could enter the technical and content contact information directly as literals in the mapping editor dialog (assuming they’re the same for all records). However, we strongly discourage you from doing this. For one thing, this is just bad style, since it will scatter the information published over different places: Some will be retrieved from the database, some (the literals) will be loaded from the configuration files. Changing the metadata of your service later would then require access to the database and access to the BioCASe configuration – that means the configuration tool password. Be nice to the person who will do these changes and put everything neatly in a database table! The second reason is even more important: Using literals for an element will result in this element to become not searchable, which means it cannot be used in a filter of a Search or a Scan request. Even though this behaviour is discoverable with a Capabilities request (by evaluating the ''searchable'' flag), a potential user of your web service might just not think of that, so it’s a good idea to avoid that in first place. Please use literals sparsely! In the mapping editor window, you can select several source columns and several literals, even a combination of both. The Provider Software will simply concatenate the values retrieved from the database and the literals specified (in the order they’re given). But again: Keep in mind that this will make the element not searchable. A better solution is to do any concatenation operations in a database view. Use the mapping editor window to add mappings for all eight mandatory fields: Click on the <strong>+</strong> symbol next to the element, choose table/column name and the correct data type, then press <small>Save</small>. When you’re done, the page should look similar to the following image. Press <small>Save</small> in the main window to write your changes to the configuration files. [[File:abcdMappingEditorMandatoryElements.png]] == Initial Testing of the Data Source == Once you’ve mapped all mandatory elements, you can do the first test of your BioCASe web service. Even though you haven’t mapped any real information yet, it is a good idea to check the service and solve any problems hitherto before continuing with the mapping. Click on <small>Test mapping!</small> to open the Query Form that can be used for this purpose: [[File:abcdQueryFormEmpty.png]] With the Query Form you can send a BioCASe request to your web service; it will wait for the reply of the web service and display the response document. On top of the page you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ''flora'' being the name of the datasource. Make sure this is correct for your installation! Into the text box below you can enter the request XML text. Because it would be cumbersome to enter the whole BioCASe request, you can use the links below the box to load templates. For now, click on <small>ABCD2 Search</small> to fill the box with a BioCASe Search request for documents in the ABCD2 format (the one we’ve just mapped): <pre> <?xml version='1.0' encoding='UTF-8'?> <request xmlns='http://www.biocase.org/schemas/protocol/1.3'> <header><type>search</type></header> <search> <requestFormat>http://www.tdwg.org/schemas/abcd/2.06</requestFormat> <responseFormat start='0' limit='10'>http://www.tdwg.org/schemas/abcd/2.06</responseFormat> <filter> <like path='/DataSets/DataSet/Units/Unit/Identifications/Identification/Result/ TaxonIdentified/ScientificName/FullScientificNameString'>A*</like> </filter> <count>false</count> </search> </request> </pre> As you can see, there is a filter set on the ''ScientificName'' element. Since we haven’t mapped this element yet, this won’t work. So let’s remove the filter (all lines between <filter> and </filter>), leaving the text box like this: [[File:abcdQueryFormFilled.png]] Pressing <small>Submit</small> will send the request to the web service and – hopefully – display the response document with the correct ABCD records. For the mapping we’ve just created, it should look similar to the following document (use the <strong>–</strong> symbol next to the trees TechnicalContact, ContentContact and Metadata to collapse them): [[File:abcdResponseMandatoryElements.png]] If you get an error, please check out the ___debugging tutorial__ on how to debug a web service. You should continue the mapping process only after your web service returns results for the basal mapping we’ve just created. == Adding additional ABCD Elements == Once you’ve successfully managed to map the mandatory ABCD elements (“successfully” means the web service returns ABCD documents), you can go on with the mapping process. This is usually an iterative process: You add a handful of elements (for example, country, country code, locality text and coordinates for the gathering site), then you use the Query Form to see if the information ends up where it is supposed to do. This is the reason the Query Form opens up in a separate tab (or window, of you configured your web browser so): In one tab, you use the mapping editor to add new elements; after pressing Save you go to the Query Form tab and see what the response documents look like. Redo this until you’ve mapped all information you want to publish. In order to map new ABCD concepts, tick the check box labelled <small>Show all concepts</small> and press <small>Refresh</small>. The mapping editor will then display the whole ABCD tree, which looks admittedly a bit intimidating. In order to find the concept you’re looking for, use the search function of your browser. If you want to map the country name of the gathering site, for example, hit <small>Ctrl + F</small> (for Firefox, Microsoft IE and Opera) and type in the name of the ABCD element, namely <small>country</small>. The browser will jump to the Country tree of the Gathering element, where you’ll find the elements for country name (Name), ISO country code (ISO3166Code) and the country name in the country’s language (NameDerived): Continue the sequence mapping new elements/testing until you’ve mapped all information you want to publish. You can have a look at the list of recommend and commonly used ABCD elements here. The full ABCD documentation can be found here. Here you can have a look at a sample response ABCD document for a web service with 77 concepts mapped. Of course your documents will look different, because you will probably use different ABCD elements. But still it’s good to get an idea: asdfads 450ff4724f5d3a6d24b9602619b1baf990c9f46c 90 86 2011-06-30T10:51:21Z JoergHoletschek 3 /* Adding additional ABCD Elements */ wikitext text/x-wiki This tutorial will explain how to set up a mapping for the ABCD2.06 data schema. If you intend to map another schema (e.g. one of the ABCD extension ABCD-EFG, ABCD-DNA or HISPID), you still should take the time to read it. Large portions of the process will be identical. == Access to Biological Collection Data Schema (ABCD) == ABCD in its current version 2.06 is the data schema typically used in conjunction with BioCASe. It is a highly complex XML data schema with about 1,000 elements that is able to store almost every piece of information that can be found in a natural history specimen collection or an observation database. The complexity is due to the fact that it can be used for a wide range of collections/databases – for living and preserved specimens, observations and culture collections, for zoological, botanical, bacterial and viral collections, marine or terrestrial, for herbaria, botanic and zoological gardens. For each of these special types, ABCD features special sections where information specific for this type can be stored. Other sections of ABCD will be shared by all these types, for example for gathering site, identifier or metadata. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a list of commonly used ABCD elements on the Wiki page [[CommonABCD2Concepts]]. == Creating an ABCD Mapping for a Data Source == If you created a data source from a template (e.g. ''abcdmetadata''), a schema mapping for ABCD will be already existent. In this case, you can directly jump to the next section. If you created an empty data source, the section <small>Schemas</small> in the data source configuration overview will be empty: [[File:abcdSchemaMappings.png]] In order to create a new schema mapping, select the desired schema from the list and click <small>Create</small>. For ABCD, choose <small>ABCD_2.06.xml</small> (be careful not to hit one of the ABCD extensions, they look quite similar). After clicking <small>Create</small>, you will be directed to the mapping editor, which will allow you to edit the mappings. == The Mapping Editor == The mapping editor allows you to add, remove and edit ABCD mappings. On top you’ll find some summary information, on the bottom you’ll see the structure of the ABCD mapping: [[File:abcdMappingEditorEmpty.png]] First thing we will do is to set the root table, which is the table that holds the records you want to publish (specimens or observations). That table must hold one record per object to be published, identified by a unique identifier (called ''UnitID''). This identifier must not be the primary key, as long as it is unique (if that sentence confuses you, ignore it and just keep in mind that the UnitID must be unique.) Use the drop down box labelled <small>Root table Alias</small> to select the root table of your data model. If your metadata table holds just one record (because all occurrences published share the same metadata) and if it is not linked to the root table by a foreign key, you can specify this table in the drop-down box <small>Static table Alias</small>. The Provider Software will use a natural join to connect the metadata record stored in this table to all occurrences published. If your metadata table is referenced by a foreign key in the root able, you must not declare the metadata table as a static table! So either join the metadata table using a foreign key OR declare it as a static table. When you’ve set the root table (and static table, if you use that), press <small>Save</small>. As with the DB structure setup, this will write your changes to the configuration files, closing the tab without saving will discard your changes! Also, in case you messed something up, you can use the <small>Revert</small> button to restore the state when you’ve saved for the last time. In the mapping editor, the link <small>Overview</small> at the top of the page will take you back to the datasource configuration overview. After creating a mapping, it will appear in the section <small>Schemas</small>. The entry will show the number of mapped elements (0 for now) and the schema namespace. Pressing the trash can symbol will move it to the trash can, which means that your web service does not support this schema anymore; if you did this by mistake, you can restore it by pressing the <small>Restore</small> button. <small>Purge</small> will delete the schema mapping permanently: [[File:abcdSchemaMappings2.png]] You can map several schemas for a given datasource. For example, in addition to the current ABCD2 standard, you could add the deprecated version 1.2 (even though I have no idea why you should do that) or DarwinCore. Just select the appropriate schema in the drop-down list and click <small>Create</small>. Since the namespace of a schema will be used in requests, you cannot create two mappings for the same namespace. Trying this will get you an error message. == Adding Mandatory ABCD Elements == Clicking on a schema in the datasource overview page will take you again to the mapping editor. In the lower half of the page, you will see the basic structure of an ABCD document: [[File:abcdMappingEditorEmptyBottom.png]] <strong>Technical/Content Contact:</strong> These trees hold information on who should be contacted for technical issues or questions concerning the published database, each with name, address, email address and telephone number. <strong>Metadata:</strong> In this sub tree you’ll find all metadata fields, for example title and description of the dataset, ownership information, date last modified, IPR statements etc. <strong>Units:</strong> This part stores the "real" data you want to publish, that means all specimen or observation data. The three elements already shown in the tree are identifiers that make up the so called "Triple ID", which uniquely identifies records within primary biodiversity networks such as BioCASe and GBIF: * ''UnitID'', which is a unique identifier for each record to be published (you can use the catalogue number, for example, or an existing barcode); * ''SourceInstitutionID'', an identifier for the institution publishing the dataset (for example BGBM); * ''SourceID'', an identifier for the dataset (e.g. “Herbarium Berolinense’’). The elements already shown in the mapping editor are printed in red because they are mandatory, which means they are required for an ABCD document. Without them being mapped (to database fields that are non-empty), the Provider Software will not be able to construct valid ABCD documents and will not publish any records. So the first elements that should be mapped are the mandatory fields. To add a mapping, click the <strong>+</strong> symbol next to the field, which will open the mapping editor dialog, where you can select the database table/column you want to map the ABCD element to: [[File:abcdMappingEditorDialog.png]] In this case, the element ''/DataSets/DataSet/Metadata/RevisionData/DateModified'' is mapped to the column ''source_update'' of the ''metadata'' table, which has the data type ''date''. (Remember that table/column retrieval is not supported for all DBMS, so if you’re unlucky, you might need to type in table/column name manually.) <small>Save</small> will accept your settings and close the dialog, <small>Cancel</small> will discard any changes. Instead of mapping an element to a database field, you can map it to a literal. For example, you could enter the technical and content contact information directly as literals in the mapping editor dialog (assuming they’re the same for all records). However, we strongly discourage you from doing this. For one thing, this is just bad style, since it will scatter the information published over different places: Some will be retrieved from the database, some (the literals) will be loaded from the configuration files. Changing the metadata of your service later would then require access to the database and access to the BioCASe configuration – that means the configuration tool password. Be nice to the person who will do these changes and put everything neatly in a database table! The second reason is even more important: Using literals for an element will result in this element to become not searchable, which means it cannot be used in a filter of a Search or a Scan request. Even though this behaviour is discoverable with a Capabilities request (by evaluating the ''searchable'' flag), a potential user of your web service might just not think of that, so it’s a good idea to avoid that in first place. Please use literals sparsely! In the mapping editor window, you can select several source columns and several literals, even a combination of both. The Provider Software will simply concatenate the values retrieved from the database and the literals specified (in the order they’re given). But again: Keep in mind that this will make the element not searchable. A better solution is to do any concatenation operations in a database view. Use the mapping editor window to add mappings for all eight mandatory fields: Click on the <strong>+</strong> symbol next to the element, choose table/column name and the correct data type, then press <small>Save</small>. When you’re done, the page should look similar to the following image. Press <small>Save</small> in the main window to write your changes to the configuration files. [[File:abcdMappingEditorMandatoryElements.png]] == Initial Testing of the Data Source == Once you’ve mapped all mandatory elements, you can do the first test of your BioCASe web service. Even though you haven’t mapped any real information yet, it is a good idea to check the service and solve any problems hitherto before continuing with the mapping. Click on <small>Test mapping!</small> to open the Query Form that can be used for this purpose: [[File:abcdQueryFormEmpty.png]] With the Query Form you can send a BioCASe request to your web service; it will wait for the reply of the web service and display the response document. On top of the page you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ''flora'' being the name of the datasource. Make sure this is correct for your installation! Into the text box below you can enter the request XML text. Because it would be cumbersome to enter the whole BioCASe request, you can use the links below the box to load templates. For now, click on <small>ABCD2 Search</small> to fill the box with a BioCASe Search request for documents in the ABCD2 format (the one we’ve just mapped): <pre> <?xml version='1.0' encoding='UTF-8'?> <request xmlns='http://www.biocase.org/schemas/protocol/1.3'> <header><type>search</type></header> <search> <requestFormat>http://www.tdwg.org/schemas/abcd/2.06</requestFormat> <responseFormat start='0' limit='10'>http://www.tdwg.org/schemas/abcd/2.06</responseFormat> <filter> <like path='/DataSets/DataSet/Units/Unit/Identifications/Identification/Result/ TaxonIdentified/ScientificName/FullScientificNameString'>A*</like> </filter> <count>false</count> </search> </request> </pre> As you can see, there is a filter set on the ''ScientificName'' element. Since we haven’t mapped this element yet, this won’t work. So let’s remove the filter (all lines between <filter> and </filter>), leaving the text box like this: [[File:abcdQueryFormFilled.png]] Pressing <small>Submit</small> will send the request to the web service and – hopefully – display the response document with the correct ABCD records. For the mapping we’ve just created, it should look similar to the following document (use the <strong>–</strong> symbol next to the trees TechnicalContact, ContentContact and Metadata to collapse them): [[File:abcdResponseMandatoryElements.png]] If you get an error, please check out the ___debugging tutorial__ on how to debug a web service. You should continue the mapping process only after your web service returns results for the basal mapping we’ve just created. == Adding additional ABCD Elements == Once you’ve successfully managed to map the mandatory ABCD elements ("successfully" means the web service returns ABCD documents), you can go on with the mapping process. This is usually an iterative process: You add a handful of elements (for example, country, country code, locality text and coordinates for the gathering site), then you use the Query Form to see if the information ends up where it is supposed to do. This is the reason the Query Form opens up in a separate tab (or window, of you configured your web browser so): In one tab, you use the mapping editor to add new elements; after pressing <small>Save</small> you go to the Query Form tab and see what the response documents look like. Redo this until you’ve mapped all information you want to publish. In order to map new ABCD concepts, tick the check box labelled <small>Show all concepts</small> and press <small>Refresh</small>. The mapping editor will then display the whole ABCD tree, which looks admittedly a bit intimidating. In order to find the concept you’re looking for, use the search function of your browser. If you want to map the country name of the gathering site, for example, hit <small>Ctrl + F</small> (for Firefox, Microsoft IE and Opera) and type in the name of the ABCD element, namely ''country''. The browser will jump to the ''Country'' tree of the ''Gathering'' element, where you’ll find the elements for country name (''Name''), ISO country code (''ISO3166Code'') and the country name in the country’s language (''NameDerived''): [[File:abcdMappingEditorSearch.png]] Continue the sequence mapping new elements/testing until you’ve mapped all information you want to publish. You can have a look at the list of recommend and commonly used ABCD elements on the [[CommonABCD2Concepts]] page. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki]. The following link will show you a sample response ABCD document for a web service with 77 concepts mapped. Of course your documents will look different, because you will probably use different ABCD elements. But still it’s good to get an idea: [[File:abcdDocumentFullMapping.xml]] bf7946cdb5bba6b74361c48788e4d2a4a817d538 92 90 2011-06-30T11:00:04Z JoergHoletschek 3 /* Adding additional ABCD Elements */ wikitext text/x-wiki This tutorial will explain how to set up a mapping for the ABCD2.06 data schema. If you intend to map another schema (e.g. one of the ABCD extension ABCD-EFG, ABCD-DNA or HISPID), you still should take the time to read it. Large portions of the process will be identical. == Access to Biological Collection Data Schema (ABCD) == ABCD in its current version 2.06 is the data schema typically used in conjunction with BioCASe. It is a highly complex XML data schema with about 1,000 elements that is able to store almost every piece of information that can be found in a natural history specimen collection or an observation database. The complexity is due to the fact that it can be used for a wide range of collections/databases – for living and preserved specimens, observations and culture collections, for zoological, botanical, bacterial and viral collections, marine or terrestrial, for herbaria, botanic and zoological gardens. For each of these special types, ABCD features special sections where information specific for this type can be stored. Other sections of ABCD will be shared by all these types, for example for gathering site, identifier or metadata. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a list of commonly used ABCD elements on the Wiki page [[CommonABCD2Concepts]]. == Creating an ABCD Mapping for a Data Source == If you created a data source from a template (e.g. ''abcdmetadata''), a schema mapping for ABCD will be already existent. In this case, you can directly jump to the next section. If you created an empty data source, the section <small>Schemas</small> in the data source configuration overview will be empty: [[File:abcdSchemaMappings.png]] In order to create a new schema mapping, select the desired schema from the list and click <small>Create</small>. For ABCD, choose <small>ABCD_2.06.xml</small> (be careful not to hit one of the ABCD extensions, they look quite similar). After clicking <small>Create</small>, you will be directed to the mapping editor, which will allow you to edit the mappings. == The Mapping Editor == The mapping editor allows you to add, remove and edit ABCD mappings. On top you’ll find some summary information, on the bottom you’ll see the structure of the ABCD mapping: [[File:abcdMappingEditorEmpty.png]] First thing we will do is to set the root table, which is the table that holds the records you want to publish (specimens or observations). That table must hold one record per object to be published, identified by a unique identifier (called ''UnitID''). This identifier must not be the primary key, as long as it is unique (if that sentence confuses you, ignore it and just keep in mind that the UnitID must be unique.) Use the drop down box labelled <small>Root table Alias</small> to select the root table of your data model. If your metadata table holds just one record (because all occurrences published share the same metadata) and if it is not linked to the root table by a foreign key, you can specify this table in the drop-down box <small>Static table Alias</small>. The Provider Software will use a natural join to connect the metadata record stored in this table to all occurrences published. If your metadata table is referenced by a foreign key in the root able, you must not declare the metadata table as a static table! So either join the metadata table using a foreign key OR declare it as a static table. When you’ve set the root table (and static table, if you use that), press <small>Save</small>. As with the DB structure setup, this will write your changes to the configuration files, closing the tab without saving will discard your changes! Also, in case you messed something up, you can use the <small>Revert</small> button to restore the state when you’ve saved for the last time. In the mapping editor, the link <small>Overview</small> at the top of the page will take you back to the datasource configuration overview. After creating a mapping, it will appear in the section <small>Schemas</small>. The entry will show the number of mapped elements (0 for now) and the schema namespace. Pressing the trash can symbol will move it to the trash can, which means that your web service does not support this schema anymore; if you did this by mistake, you can restore it by pressing the <small>Restore</small> button. <small>Purge</small> will delete the schema mapping permanently: [[File:abcdSchemaMappings2.png]] You can map several schemas for a given datasource. For example, in addition to the current ABCD2 standard, you could add the deprecated version 1.2 (even though I have no idea why you should do that) or DarwinCore. Just select the appropriate schema in the drop-down list and click <small>Create</small>. Since the namespace of a schema will be used in requests, you cannot create two mappings for the same namespace. Trying this will get you an error message. == Adding Mandatory ABCD Elements == Clicking on a schema in the datasource overview page will take you again to the mapping editor. In the lower half of the page, you will see the basic structure of an ABCD document: [[File:abcdMappingEditorEmptyBottom.png]] <strong>Technical/Content Contact:</strong> These trees hold information on who should be contacted for technical issues or questions concerning the published database, each with name, address, email address and telephone number. <strong>Metadata:</strong> In this sub tree you’ll find all metadata fields, for example title and description of the dataset, ownership information, date last modified, IPR statements etc. <strong>Units:</strong> This part stores the "real" data you want to publish, that means all specimen or observation data. The three elements already shown in the tree are identifiers that make up the so called "Triple ID", which uniquely identifies records within primary biodiversity networks such as BioCASe and GBIF: * ''UnitID'', which is a unique identifier for each record to be published (you can use the catalogue number, for example, or an existing barcode); * ''SourceInstitutionID'', an identifier for the institution publishing the dataset (for example BGBM); * ''SourceID'', an identifier for the dataset (e.g. “Herbarium Berolinense’’). The elements already shown in the mapping editor are printed in red because they are mandatory, which means they are required for an ABCD document. Without them being mapped (to database fields that are non-empty), the Provider Software will not be able to construct valid ABCD documents and will not publish any records. So the first elements that should be mapped are the mandatory fields. To add a mapping, click the <strong>+</strong> symbol next to the field, which will open the mapping editor dialog, where you can select the database table/column you want to map the ABCD element to: [[File:abcdMappingEditorDialog.png]] In this case, the element ''/DataSets/DataSet/Metadata/RevisionData/DateModified'' is mapped to the column ''source_update'' of the ''metadata'' table, which has the data type ''date''. (Remember that table/column retrieval is not supported for all DBMS, so if you’re unlucky, you might need to type in table/column name manually.) <small>Save</small> will accept your settings and close the dialog, <small>Cancel</small> will discard any changes. Instead of mapping an element to a database field, you can map it to a literal. For example, you could enter the technical and content contact information directly as literals in the mapping editor dialog (assuming they’re the same for all records). However, we strongly discourage you from doing this. For one thing, this is just bad style, since it will scatter the information published over different places: Some will be retrieved from the database, some (the literals) will be loaded from the configuration files. Changing the metadata of your service later would then require access to the database and access to the BioCASe configuration – that means the configuration tool password. Be nice to the person who will do these changes and put everything neatly in a database table! The second reason is even more important: Using literals for an element will result in this element to become not searchable, which means it cannot be used in a filter of a Search or a Scan request. Even though this behaviour is discoverable with a Capabilities request (by evaluating the ''searchable'' flag), a potential user of your web service might just not think of that, so it’s a good idea to avoid that in first place. Please use literals sparsely! In the mapping editor window, you can select several source columns and several literals, even a combination of both. The Provider Software will simply concatenate the values retrieved from the database and the literals specified (in the order they’re given). But again: Keep in mind that this will make the element not searchable. A better solution is to do any concatenation operations in a database view. Use the mapping editor window to add mappings for all eight mandatory fields: Click on the <strong>+</strong> symbol next to the element, choose table/column name and the correct data type, then press <small>Save</small>. When you’re done, the page should look similar to the following image. Press <small>Save</small> in the main window to write your changes to the configuration files. [[File:abcdMappingEditorMandatoryElements.png]] == Initial Testing of the Data Source == Once you’ve mapped all mandatory elements, you can do the first test of your BioCASe web service. Even though you haven’t mapped any real information yet, it is a good idea to check the service and solve any problems hitherto before continuing with the mapping. Click on <small>Test mapping!</small> to open the Query Form that can be used for this purpose: [[File:abcdQueryFormEmpty.png]] With the Query Form you can send a BioCASe request to your web service; it will wait for the reply of the web service and display the response document. On top of the page you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ''flora'' being the name of the datasource. Make sure this is correct for your installation! Into the text box below you can enter the request XML text. Because it would be cumbersome to enter the whole BioCASe request, you can use the links below the box to load templates. For now, click on <small>ABCD2 Search</small> to fill the box with a BioCASe Search request for documents in the ABCD2 format (the one we’ve just mapped): <pre> <?xml version='1.0' encoding='UTF-8'?> <request xmlns='http://www.biocase.org/schemas/protocol/1.3'> <header><type>search</type></header> <search> <requestFormat>http://www.tdwg.org/schemas/abcd/2.06</requestFormat> <responseFormat start='0' limit='10'>http://www.tdwg.org/schemas/abcd/2.06</responseFormat> <filter> <like path='/DataSets/DataSet/Units/Unit/Identifications/Identification/Result/ TaxonIdentified/ScientificName/FullScientificNameString'>A*</like> </filter> <count>false</count> </search> </request> </pre> As you can see, there is a filter set on the ''ScientificName'' element. Since we haven’t mapped this element yet, this won’t work. So let’s remove the filter (all lines between <filter> and </filter>), leaving the text box like this: [[File:abcdQueryFormFilled.png]] Pressing <small>Submit</small> will send the request to the web service and – hopefully – display the response document with the correct ABCD records. For the mapping we’ve just created, it should look similar to the following document (use the <strong>–</strong> symbol next to the trees TechnicalContact, ContentContact and Metadata to collapse them): [[File:abcdResponseMandatoryElements.png]] If you get an error, please check out the ___debugging tutorial__ on how to debug a web service. You should continue the mapping process only after your web service returns results for the basal mapping we’ve just created. == Adding additional ABCD Elements == Once you’ve successfully managed to map the mandatory ABCD elements ("successfully" means the web service returns ABCD documents), you can go on with the mapping process. This is usually an iterative process: You add a handful of elements (for example, country, country code, locality text and coordinates for the gathering site), then you use the Query Form to see if the information ends up where it is supposed to do. This is the reason the Query Form opens up in a separate tab (or window, of you configured your web browser so): In one tab, you use the mapping editor to add new elements; after pressing <small>Save</small> you go to the Query Form tab and see what the response documents look like. Redo this until you’ve mapped all information you want to publish. In order to map new ABCD concepts, tick the check box labelled <small>Show all concepts</small> and press <small>Refresh</small>. The mapping editor will then display the whole ABCD tree, which looks admittedly a bit intimidating. In order to find the concept you’re looking for, use the search function of your browser. If you want to map the country name of the gathering site, for example, hit <small>Ctrl + F</small> (for Firefox, Microsoft IE and Opera) and type in the name of the ABCD element, namely ''country''. The browser will jump to the ''Country'' tree of the ''Gathering'' element, where you’ll find the elements for country name (''Name''), ISO country code (''ISO3166Code'') and the country name in the country’s language (''NameDerived''): [[File:abcdMappingEditorSearch.png|500px]] Continue the sequence mapping new elements/testing until you’ve mapped all information you want to publish. You can have a look at the list of recommend and commonly used ABCD elements on the [[CommonABCD2Concepts]] page. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki]. The following page will show you a sample response ABCD document for a web service with 77 concepts mapped. Of course your documents will look different, because you will probably use different ABCD elements, but still it’s good to get an idea. For easier viewing, you can save the contents of the box into a file with the extension .xml and open that in Firefix or MS Internet Explorer (don't use Safari). You will then be able to collapse the different XML sub trees: [[SampleABCDDocument]]. fe66360e5a4e16b9ad83ead889b23a9664bb58f5 6 30 78 2011-06-30T10:27:03Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 31 80 2011-06-30T10:30:47Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 32 81 2011-06-30T10:31:05Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 33 83 2011-06-30T10:37:08Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 34 84 2011-06-30T10:37:22Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 35 85 2011-06-30T10:37:39Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 36 87 2011-06-30T10:44:30Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 37 88 2011-06-30T10:44:59Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 38 89 2011-06-30T10:45:21Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 39 91 2011-06-30T10:51:38Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 0 40 93 2011-06-30T11:02:02Z JoergHoletschek 3 Created page with "This page shows a sample response ABCD2 document for a web service with 77 concepts mapped. For easier viewing, you can save the contents of the box into a file with the extensio..." wikitext text/x-wiki This page shows a sample response ABCD2 document for a web service with 77 concepts mapped. For easier viewing, you can save the contents of the box into a file with the extension .xml and open that in Firefix or MS Internet Explorer (don't use Safari). You will then be able to collapse the different XML sub trees. <pre> <?xml version='1.0' encoding='UTF-8'?> <biocase:response xmlns:abcd="http://www.tdwg.org/schemas/abcd/2.06" xmlns:biocase="http://www.biocase.org/schemas/protocol/1.3" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.biocase.org/schemas/protocol/1.3 http://www.bgbm.org/biodivinf/schema/protocol_1_31.xsd"> <!-- XML generated by BioCASE PyWrapper software version 2.6.0. Made in Berlin. --> <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> <biocase:content recordCount="10" recordDropped="0" recordStart="0" totalSearchHits="11"> <abcd:DataSets> <abcd:DataSet> <abcd:TechnicalContacts> <abcd:TechnicalContact> <abcd:Name>Botanic Garden and Botanical Museum Berlin-Dahlem</abcd:Name> <abcd:Email>biodiversitydata@bgbm.org</abcd:Email> <abcd:Address>Königin-Luise-Str. 6-8, D-14195 Berlin</abcd:Address> </abcd:TechnicalContact> </abcd:TechnicalContacts> <abcd:ContentContacts> <abcd:ContentContact> <abcd:Name>Dr. Monika Engels</abcd:Name> <abcd:Email>engels@botanik.uni-hamburg.de</abcd:Email> <abcd:Phone>+49 40 428 16 321</abcd:Phone> <abcd:Address>Ohnhorststr. 18, D-22609 Hamburg</abcd:Address> </abcd:ContentContact> </abcd:ContentContacts> <abcd:Metadata> <abcd:Description> <abcd:Representation abcd:language="EN"> <abcd:Title>Desmidiaceae Engels</abcd:Title> <abcd:Details>A new record of reference material of the Desmidiaceae of Germany was compiled, using the databank “Specify”. Aim of the project was to summarize and digitalize the data of the dried and otherwise preserved specimens stored in the German herbaria as well as of the living strains, cultured in culture collections. Many of the references are completed by figures of the labels or by scanned micrographs from preparations for the light- or scanning microscopes.</abcd:Details> <abcd:URI>http://www.gbif.de/botanik/projekte/desmidiaceae_hamburg_ffm/index_html</abcd:URI> </abcd:Representation> </abcd:Description> <abcd:IconURI>http://ww3.bgbm.org/providerResources/BzF.jpg</abcd:IconURI> <abcd:RevisionData> <abcd:Creators>Desmidiaceae Engels (BzF)</abcd:Creators> <abcd:DateModified>2008-01-23T00:00:00</abcd:DateModified> </abcd:RevisionData> <abcd:Owners> <abcd:Owner> <abcd:Organisation> <abcd:Name> <abcd:Representation abcd:language="EN"> <abcd:Text>Biozentrum Klein-Flottbek, Universität Hamburg</abcd:Text> <abcd:Abbreviation>BzF</abcd:Abbreviation> </abcd:Representation> </abcd:Name> </abcd:Organisation> </abcd:Owner> </abcd:Owners> <abcd:IPRStatements> <abcd:IPRDeclarations> <abcd:IPRDeclaration abcd:language="EN"> <abcd:Text>The Intellectual Property Rights are held by the legal owner or, in case of living persons, by the collector or determinator.</abcd:Text> </abcd:IPRDeclaration> </abcd:IPRDeclarations> <abcd:Copyrights> <abcd:Copyright abcd:language="EN"> <abcd:Text>No part of this data base may be copied or reproduced without written permission from the legal owner.</abcd:Text> </abcd:Copyright> </abcd:Copyrights> <abcd:TermsOfUseStatements> <abcd:TermsOfUse abcd:language="EN"> <abcd:Text>The use of the data is allowed only for non-profit scientific use and for non-profit nature conservation purpose. The database or part of it may only be used or copied by the written permission from the legal owner. If used for publication, we ask for a copy or an off-print.</abcd:Text> </abcd:TermsOfUse> </abcd:TermsOfUseStatements> <abcd:Disclaimers> <abcd:Disclaimer abcd:language="EN"> <abcd:Text>No responsibility is accepted for the accuracy of the information in this database.</abcd:Text> </abcd:Disclaimer> </abcd:Disclaimers> <abcd:Acknowledgements> <abcd:Acknowledgement abcd:language="EN"> <abcd:Text>The authors are much indebted to the Scientists of the Informatics Biodiversity Research Center der University of Kansas, USA (http://www.specifysoftware.org), who developed the ?Specify?-Software. Many thanks also to the curators of different Herbaria: Dr. Regine Jahn (BGBM, FU-Berlin), Dr. Dagmar Triebel (Botanische Staatssammlung München), Dr. Monika Steinhof (Übersee Museum Bremen), Christian Blümel (Universität Rostock) and Dr. Christian Printzen (Forschungsinstitut Senckenberg Frankfurt a.M.). Last not least thanks to the busy co-workers Sebastian Frerichmann, Maas Mollenhauer, Constanze Kovaci and Petra Kretschmann.</abcd:Text> </abcd:Acknowledgement> </abcd:Acknowledgements> <abcd:Citations> <abcd:Citation abcd:language="EN"> <abcd:Text>Engels, Monika 2003 - (continuously updated) Catalogus novus et amplificatus speciminum et viventium algarum Desmidiacearum (New and extended catalogue of herbarium specimen and living material of Desmidiaceae in Germany).</abcd:Text> </abcd:Citation> </abcd:Citations> </abcd:IPRStatements> </abcd:Metadata> <abcd:Units> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-6239</abcd:UnitID> <abcd:UnitReferences> <abcd:UnitReference> <abcd:TitleCitation>Algae aquae dulcis exsiccatae praecique scandinavicae, quas adjectis algis marinis chlorophyllaceis et phycochromaceis distribuerunt., Stockholm 1877-1889, Vol. 1-26</abcd:TitleCitation> </abcd:UnitReference> </abcd:UnitReferences> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Mesotaeniaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Zygnematales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Ancylonema nordenskiöldii</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:Identifiers> <abcd:Identifier> <abcd:PersonName> <abcd:FullName>Nordstedt, O.</abcd:FullName> </abcd:PersonName> </abcd:Identifier> </abcd:Identifiers> <abcd:Date> <abcd:DateText>1878-07-18</abcd:DateText> </abcd:Date> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>Specimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:MultiMediaObjects> <abcd:MultiMediaObject> <abcd:FileURI>http://ww2.bgbm.org/herbarium/Externe_Bildsammlungen/AlgenEngels/L-Scans-B-Konv/6239.jpg</abcd:FileURI> </abcd:MultiMediaObject> </abcd:MultiMediaObjects> <abcd:Gathering> <abcd:DateTime>1878<abcd:DateText>1878</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Nordstedt, O.</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Brixdal</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>NO</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Nordfjord</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:AreaDetail language="DE">see label</abcd:AreaDetail> <abcd:Biotope> <abcd:Name language="EN">other</abcd:Name> <abcd:Text language="DE">see label</abcd:Text> </abcd:Biotope> <abcd:Notes language="EN">Europe</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: B 4000 08107; Specimen Location: B.</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-3397</abcd:UnitID> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Actinotaenium cucurbita (Brébisson) Teiling ex Ruzicka et Pouzar 1978</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 108, tab. 17: 7a-e as Cosmarium cucurbita; taxon citation: Teiling, E. 1954: Actinotaenium genus Desmidiacearum resuscitatum. - Bot. Notiser 100, p. 406, fig. 66 (invalidy pblished). Validation: Ruzicka, J. &amp; Pouzar, Z. 1978:Erwägungen über die Taxonomie und Nomenklatur der Gattung Actinotaenium TEIL. - Folia Geobot. Phytotax., Praha, 13: p.44</abcd:Notes> </abcd:Identification> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Cosmarium cucurbita (Bréb.) Ralfs 1848</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>false</abcd:PreferredFlag> <abcd:Identifiers> <abcd:Identifier> <abcd:PersonName> <abcd:FullName>Siegmund, W.</abcd:FullName> </abcd:PersonName> </abcd:Identifier> </abcd:Identifiers> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Date> <abcd:DateText>1872-07</abcd:DateText> </abcd:Date> <abcd:Notes>p. 108, tab. 17: 7a-e</abcd:Notes> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>Specimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:MultiMediaObjects> <abcd:MultiMediaObject> <abcd:FileURI>http://ww2.bgbm.org/herbarium/Externe_Bildsammlungen/AlgenEngels/L-Scans-HBG-Konv/3397.jpg</abcd:FileURI> </abcd:MultiMediaObject> </abcd:MultiMediaObjects> <abcd:Gathering> <abcd:DateTime>1872<abcd:DateText>1872</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Siegmund, W.</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Reichenberg</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>DE</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Sachsen</abcd:AreaName> </abcd:NamedArea> <abcd:NamedArea> <abcd:AreaName>Dresden</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:AreaDetail language="DE">bei Reichenberg, see label</abcd:AreaDetail> <abcd:Biotope> <abcd:Name language="EN">other</abcd:Name> <abcd:Text language="DE">see label</abcd:Text> </abcd:Biotope> <abcd:Notes language="EN">Europe</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: 8186; Specimen Location: HBG.</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-2483</abcd:UnitID> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Cosmarium turgidum Bréb. 1848</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>false</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 110, tab. 32: 8</abcd:Notes> </abcd:Identification> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Actinotaenium turgidum (Brébisson) Teiling 1954 ex Ruzicka et Pouzar 1978</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>as Cosmarium turgidum p. 110, tab. 32: 8; taxon citation: invalidly published as A. turgidum (Bréb.) Teiling, E. 1954: Actinotaenium genus Desmidiacearum resuscitatum. - Bot. Notiser 100 - p.395, fig.21; Validation by R?ži?ka &amp; Pouzar 1978:Erwägungen über die Taxonomie und Nomenklatur der Gattung Actinotaenium TEIL. - Folia Geobot. Phytotax., Praha, 13: 13 - 66 p.60</abcd:Notes> </abcd:Identification> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Pleurotaenium turgidum De Bary 1858</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>false</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>De Bary, A: Untersuchungen über die Familie der Conjugaten (Zygnemeen und Desmidieen)., Leipzig 1858</abcd:TitleCitation> <abcd:CitationDetail>p. 91</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 75, tab. 5: 31; invalidly published</abcd:Notes> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>Specimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:Gathering> <abcd:DateTime>0<abcd:DateText>0</abcd:DateText> </abcd:DateTime> <abcd:LocalityText language="DE">Neukirch</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>DE</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Sachsen</abcd:AreaName> </abcd:NamedArea> <abcd:NamedArea> <abcd:AreaName>Chemnitz</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:AreaDetail language="DE">Oberlausitz, Neukirch</abcd:AreaDetail> <abcd:Biotope> <abcd:Name language="EN" /> <abcd:Text language="DE" /> </abcd:Biotope> <abcd:Notes language="EN">Europe</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: 7474; Specimen Location: HBG. together with Closterium hybridum, Pl. truncatum</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-4603</abcd:UnitID> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Arthrodesmus octocornis var. major Roy &amp; Biss. 1894</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:Identifiers> <abcd:Identifier> <abcd:PersonName> <abcd:FullName>Richter, Paul</abcd:FullName> </abcd:PersonName> </abcd:Identifier> </abcd:Identifiers> <abcd:Notes>see label</abcd:Notes> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>Specimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:MultiMediaObjects> <abcd:MultiMediaObject> <abcd:FileURI>http://ww2.bgbm.org/herbarium/Externe_Bildsammlungen/AlgenEngels/L-Scans-HBG-Konv/4602.jpg</abcd:FileURI> </abcd:MultiMediaObject> </abcd:MultiMediaObjects> <abcd:Gathering> <abcd:DateTime>0<abcd:DateText>0</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Richter, Paul</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Leipzig</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>DE</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Sachsen</abcd:AreaName> </abcd:NamedArea> <abcd:NamedArea> <abcd:AreaName>Leipzig</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:AreaDetail language="DE" /> <abcd:Biotope> <abcd:Name language="EN" /> <abcd:Text language="DE" /> </abcd:Biotope> <abcd:Notes language="EN">Europe</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: 8773; Specimen Location: HBG.</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-6764</abcd:UnitID> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Actinotaenium curtum var. attenuatum (Bréb.) Teil. ex Ruzicka &amp; Pouzar 1978</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>as Cosmarium attenuatum p. 110, tab. 17: 9; taxon citation: Teiling, E., 1954: Actinotaenium genus Desmidiacearum resuscitatum. - Bot. Not., Lund, 1954, p. 391, fig. 2; invaldly published; Validation: Ruzicka, J., &amp; Pouzar, Z., 1978: Erwägungen über die Taxonomie und Nomenklatur der Gattung ActinotaeniumTEIL. - Folia geobot. Phytotax., Praha, 13: 33-66., p.47</abcd:Notes> </abcd:Identification> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Cosmarium attenuatum Ralfs 1848</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>false</abcd:PreferredFlag> <abcd:Identifiers> <abcd:Identifier> <abcd:PersonName> <abcd:FullName>Richter, Paul</abcd:FullName> </abcd:PersonName> </abcd:Identifier> </abcd:Identifiers> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 110, tab. 17: 9a-d</abcd:Notes> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>Specimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:Gathering> <abcd:DateTime>1898<abcd:DateText>1898</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Rogers,</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Ennepetal-Milspe</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>DE</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Nordrhein-Westfalen</abcd:AreaName> </abcd:NamedArea> <abcd:NamedArea> <abcd:AreaName>Hagen</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:AreaDetail language="DE" /> <abcd:Biotope> <abcd:Name language="EN" /> <abcd:Text language="DE">an einer Felsenwand</abcd:Text> </abcd:Biotope> <abcd:Notes language="EN">Europe</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: B 4000 08307; Specimen Location: B.</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-3698</abcd:UnitID> <abcd:UnitReferences> <abcd:UnitReference> <abcd:TitleCitation>Algae aquae dulcis exsiccatae praecique scandinavicae, quas adjectis algis marinis chlorophyllaceis et phycochromaceis distribuerunt., Lund 1896-1903, Vol. 27-34</abcd:TitleCitation> </abcd:UnitReference> </abcd:UnitReferences> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Actinotaenium palangula (Brébisson) Teiling 1954 ex Ruzicka et Pouzar 1978</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 212; original figure: Brébisson 1856: p.132, tab.1: 21; taxon citation: Ruzicka, J. &amp; Pouzar, Z., 1978: Erwägungen über die Taxonomie und Nomenklatur der Gattung Actinotaenium TEIL. - Folia Geobot. Phytotax., Praha, 13: p.46</abcd:Notes> </abcd:Identification> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Cosmarium palangula Brébisson in Ralfs 1848</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>false</abcd:PreferredFlag> <abcd:Identifiers> <abcd:Identifier> <abcd:PersonName> <abcd:FullName>Bohlin, Knut</abcd:FullName> </abcd:PersonName> </abcd:Identifier> </abcd:Identifiers> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Date> <abcd:DateText>1898-07-06</abcd:DateText> </abcd:Date> <abcd:Notes>p. 212; original figure: Brébisson 1856: p.132, tab.1: 21</abcd:Notes> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>Specimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:MultiMediaObjects> <abcd:MultiMediaObject> <abcd:FileURI>http://ww2.bgbm.org/herbarium/Externe_Bildsammlungen/AlgenEngels/L-Scans-HBG-Konv/3690-3703.jpg</abcd:FileURI> </abcd:MultiMediaObject> </abcd:MultiMediaObjects> <abcd:Gathering> <abcd:DateTime>1898<abcd:DateText>1898</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Bohlin, Knut</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Pico da Cruz</abcd:LocalityText> <abcd:AreaDetail language="DE" /> <abcd:Biotope> <abcd:Name language="EN" /> <abcd:Text language="DE" /> </abcd:Biotope> <abcd:Notes language="EN">Africa</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: 8328; Specimen Location: HBG. sample contains: Cosmarium ralfsii var. azoricum, Cosmarium palangula, Cosmarium pyramidatum, Cosmarium venustum, Euastru didelta f. scorbiculata, Staurastrum muticum var. depressum, Staurastrum scabrum</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-4569</abcd:UnitID> <abcd:UnitReferences> <abcd:UnitReference> <abcd:TitleCitation>Die Algen Europa's (Fortsetzung der Algen Sachsens, resp. Mittel-Europa's)., Dresden 1861 - 1879, Vol. Dec. 101 - 259</abcd:TitleCitation> </abcd:UnitReference> </abcd:UnitReferences> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Staurodesmus convergens (Ehrenberg ex Ralfs) Teiling 1967</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 118, tab. 20: 3a-d; taxon citation: Teiling, E., 1948: Staurodesmus, genus novum.Containing monospinous desmids. - Bot. Not., Lund, 1948: p. 57</abcd:Notes> </abcd:Identification> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Arthrodesmus convergens Ehrenberg ex Ralfs 1848</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>false</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 118, tab. 20: 3a-d</abcd:Notes> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>Specimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:Gathering> <abcd:DateTime>0<abcd:DateText>0</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Rostock, M.</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Dretschen</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>DE</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Sachsen</abcd:AreaName> </abcd:NamedArea> <abcd:NamedArea> <abcd:AreaName>Bautzen</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:AreaDetail language="DE" /> <abcd:Biotope> <abcd:Name language="EN">peat bog</abcd:Name> <abcd:Text language="DE" /> </abcd:Biotope> <abcd:Notes language="EN">Europe</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: 8751; Specimen Location: HBG.</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-7071</abcd:UnitID> <abcd:UnitReferences> <abcd:UnitReference> <abcd:TitleCitation>Kryptogamae exsiccatae editae a museo hist. natur. vindobonensi., Wien 1894 ff</abcd:TitleCitation> </abcd:UnitReference> </abcd:UnitReferences> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Arthrodesmus quadricaudatus</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:Identifiers> <abcd:Identifier> <abcd:PersonName> <abcd:FullName>Hansgirg, A.</abcd:FullName> </abcd:PersonName> </abcd:Identifier> </abcd:Identifiers> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>Specimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:Gathering> <abcd:DateTime>0<abcd:DateText>0</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Hansgirg, A.</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Bombay</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>IN</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Bombay</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:AreaDetail language="DE" /> <abcd:Biotope> <abcd:Name language="EN">other</abcd:Name> <abcd:Text language="DE">see label</abcd:Text> </abcd:Biotope> <abcd:Notes language="EN">Asia</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: B 4000 08385; Specimen Location: B. sample contains: Cosmarium nitidulum, Arthrodesmus quadricaudatus</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-6250</abcd:UnitID> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Staurodesmus incus (Brébisson ex Ralfs) Teiling 1967</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 118, tab. 20: 4 as Arthrodesmus incus; taxon citation: Teiling, E., 1967: The desmid genus Staurodesmus. A taxonomic study. Ark. f. Bot. II, 6(11), p. p. 511, tab. 5: 9</abcd:Notes> </abcd:Identification> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Arthrodesmus incus (Bréb.) Hassall ex Ralfs 1848</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>false</abcd:PreferredFlag> <abcd:Identifiers> <abcd:Identifier> <abcd:PersonName> <abcd:FullName>Borge, O.</abcd:FullName> </abcd:PersonName> </abcd:Identifier> </abcd:Identifiers> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Date> <abcd:DateText>1889-08-08</abcd:DateText> </abcd:Date> <abcd:Notes>p. 118, tab. 20: 4a-l</abcd:Notes> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>Specimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:Gathering> <abcd:DateTime>1889<abcd:DateText>1889</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Borge, O.</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Väddö</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>SE</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Upland</abcd:AreaName> </abcd:NamedArea> <abcd:NamedArea> <abcd:AreaName>Väddö</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:AreaDetail language="DE" /> <abcd:Biotope> <abcd:Name language="EN" /> <abcd:Text language="DE" /> </abcd:Biotope> <abcd:Notes language="EN">Europe</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: B 4000 32088; Specimen Location: B.</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-6897</abcd:UnitID> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Actinotaenium cucurbita (Brébisson) Teiling ex Ruzicka et Pouzar 1978</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 108, tab. 17: 7a-e as Cosmarium cucurbita; taxon citation: Teiling, E. 1954: Actinotaenium genus Desmidiacearum resuscitatum. - Bot. Notiser 100, p. 406, fig. 66 (invalidy pblished). Validation: Ruzicka, J. &amp; Pouzar, Z. 1978:Erwägungen über die Taxonomie und Nomenklatur der Gattung Actinotaenium TEIL. - Folia Geobot. Phytotax., Praha, 13: p.44</abcd:Notes> </abcd:Identification> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Cosmarium cucurbita (Bréb.) Ralfs 1848</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>false</abcd:PreferredFlag> <abcd:Identifiers> <abcd:Identifier> <abcd:PersonName> <abcd:FullName>Bulnheim, O.</abcd:FullName> </abcd:PersonName> </abcd:Identifier> </abcd:Identifiers> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Date> <abcd:DateText>1883-07</abcd:DateText> </abcd:Date> <abcd:Notes>p. 108, tab. 17: 7a-e</abcd:Notes> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>Specimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:MultiMediaObjects> <abcd:MultiMediaObject> <abcd:FileURI>http://ww2.bgbm.org/herbarium/Externe_Bildsammlungen/AlgenEngels/L-Scans-B-Konv/6897-6899.jpg</abcd:FileURI> </abcd:MultiMediaObject> </abcd:MultiMediaObjects> <abcd:Gathering> <abcd:DateTime>1883<abcd:DateText>1883</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Bulnheim, O.</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Leipzig</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>DE</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Sachsen</abcd:AreaName> </abcd:NamedArea> <abcd:NamedArea> <abcd:AreaName>Leipzig</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:AreaDetail language="DE">see label</abcd:AreaDetail> <abcd:Biotope> <abcd:Name language="EN" /> <abcd:Text language="DE" /> </abcd:Biotope> <abcd:Notes language="EN">Europe</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: B 4000 08358; Specimen Location: B. sample contains: Cosmarium cucurbita, Euastrum didelta, Closterium intermedium</abcd:Notes> </abcd:Unit> </abcd:Units> </abcd:DataSet> </abcd:DataSets> </biocase:content> <biocase:diagnostics> <biocase:diagnostic severity="INFO">Datasource wrapper AlgenEngels requested</biocase:diagnostic> <biocase:diagnostic severity="INFO">Reading PSF from C:\Workspace\bps2\config\datasources\AlgenEngels\provider_setup_file.xml</biocase:diagnostic> <biocase:diagnostic severity="DEBUG">PSF: PSF=C:\Workspace\bps2\config\datasources\AlgenEngels\provider_setup_file.xml, recLimit=0, loglevel=40, user=WebUser, database=AlgenEngels, dbIP=, dbms=odbc_tsql, encoding=latin_1, schemas={u'http://www.tdwg.org/schemas/abcd/2.06': &lt;biocase.wrapper.psf_handler.SupportedSchema instance at 0x00EB1E90&gt;}, tablegraph=GRAPH: graph: meta, determination-taxonNames, higherTaxon-taxonNames, images-specimen, namedAreas-collectingEvent, siteFeatures-collectingEvent, specimen-determination, specimen-collectingEvent, unitReference-specimen, +++ ALIAS2TABLE: {u'collectingEvent': u'abcd_CollectingEvent', u'taxonNames': u'abcd_TaxonName', u'higherTaxon': u'abcd_HigherTaxa', u'specimen': u'abcd_Object', u'siteFeatures': u'abcd_SiteFeatureTerms', u'meta': u'metadata', u'determination': u'abcd_Determination', u'images': u'abcd_Images', u'unitReference': u'abcd_UnitReference', u'namedAreas': u'abcd_NamedAreaNames'}</biocase:diagnostic> </biocase:diagnostics> </biocase:response> </pre> 7c1d41bef46e3c01b3f116734091cd69dc75bd97 94 93 2011-06-30T11:05:01Z JoergHoletschek 3 wikitext text/x-wiki This page shows a sample response ABCD2 document for a web service with 77 concepts mapped. For easier viewing, you can save the contents of the box into a file with the extension .xml and open that in Firefix or MS Internet Explorer (don't use Safari). You will then be able to collapse the different XML sub trees. <pre> <?xml version='1.0' encoding='UTF-8'?> <biocase:response xmlns:abcd="http://www.tdwg.org/schemas/abcd/2.06" xmlns:biocase="http://www.biocase.org/schemas/protocol/1.3" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.biocase.org/schemas/protocol/1.3 http://www.bgbm.org/biodivinf/schema/protocol_1_31.xsd"> <!-- XML generated by BioCASE PyWrapper software version 2.6.0. Made in Berlin. --> <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> <biocase:content recordCount="10" recordDropped="0" recordStart="0" totalSearchHits="11"> <abcd:DataSets> <abcd:DataSet> <abcd:TechnicalContacts> <abcd:TechnicalContact> <abcd:Name>Botanic Garden and Botanical Museum Berlin-Dahlem</abcd:Name> <abcd:Email>...@...</abcd:Email> <abcd:Address>Königin-Luise-Str. 6-8, D-14195 Berlin</abcd:Address> </abcd:TechnicalContact> </abcd:TechnicalContacts> <abcd:ContentContacts> <abcd:ContentContact> <abcd:Name>Dr. ...</abcd:Name> <abcd:Email>...@...</abcd:Email> <abcd:Phone>+49 ...</abcd:Phone> <abcd:Address>..., D-22609 Hamburg</abcd:Address> </abcd:ContentContact> </abcd:ContentContacts> <abcd:Metadata> <abcd:Description> <abcd:Representation abcd:language="EN"> <abcd:Title>Desmidiaceae Engels</abcd:Title> <abcd:Details>A new record of reference material of the Desmidiaceae of Germany was compiled, using the databank “Specify”. Aim of the project was to summarize and digitalize the data of the dried and otherwise preserved specimens stored in the German herbaria as well as of the living strains, cultured in culture collections. Many of the references are completed by figures of the labels or by scanned micrographs from preparations for the light- or scanning microscopes.</abcd:Details> <abcd:URI>http://www.gbif.de/botanik/projekte/desmidiaceae_hamburg_ffm/index_html</abcd:URI> </abcd:Representation> </abcd:Description> <abcd:IconURI>http://ww3.bgbm.org/providerResources/BzF.jpg</abcd:IconURI> <abcd:RevisionData> <abcd:Creators>Desmidiaceae Engels (BzF)</abcd:Creators> <abcd:DateModified>2008-01-23T00:00:00</abcd:DateModified> </abcd:RevisionData> <abcd:Owners> <abcd:Owner> <abcd:Organisation> <abcd:Name> <abcd:Representation abcd:language="EN"> <abcd:Text>Biozentrum Klein-Flottbek, Universität Hamburg</abcd:Text> <abcd:Abbreviation>BzF</abcd:Abbreviation> </abcd:Representation> </abcd:Name> </abcd:Organisation> </abcd:Owner> </abcd:Owners> <abcd:IPRStatements> <abcd:IPRDeclarations> <abcd:IPRDeclaration abcd:language="EN"> <abcd:Text>The Intellectual Property Rights are held by the legal owner or, in case of living persons, by the collector or determinator.</abcd:Text> </abcd:IPRDeclaration> </abcd:IPRDeclarations> <abcd:Copyrights> <abcd:Copyright abcd:language="EN"> <abcd:Text>No part of this data base may be copied or reproduced without written permission from the legal owner.</abcd:Text> </abcd:Copyright> </abcd:Copyrights> <abcd:TermsOfUseStatements> <abcd:TermsOfUse abcd:language="EN"> <abcd:Text>The use of the data is allowed only for non-profit scientific use and for non-profit nature conservation purpose. The database or part of it may only be used or copied by the written permission from the legal owner. If used for publication, we ask for a copy or an off-print.</abcd:Text> </abcd:TermsOfUse> </abcd:TermsOfUseStatements> <abcd:Disclaimers> <abcd:Disclaimer abcd:language="EN"> <abcd:Text>No responsibility is accepted for the accuracy of the information in this database.</abcd:Text> </abcd:Disclaimer> </abcd:Disclaimers> <abcd:Acknowledgements> <abcd:Acknowledgement abcd:language="EN"> <abcd:Text>The authors are much indebted to the Scientists of the Informatics Biodiversity Research Center der University of Kansas, USA (http://www.specifysoftware.org), who developed the ?Specify?-Software. Many thanks also to the curators of different Herbaria: Dr. Regine Jahn (BGBM, FU-Berlin), Dr. Dagmar Triebel (Botanische Staatssammlung München), Dr. Monika Steinhof (Übersee Museum Bremen), Christian Blümel (Universität Rostock) and Dr. Christian Printzen (Forschungsinstitut Senckenberg Frankfurt a.M.). Last not least thanks to the busy co-workers Sebastian Frerichmann, Maas Mollenhauer, Constanze Kovaci and Petra Kretschmann.</abcd:Text> </abcd:Acknowledgement> </abcd:Acknowledgements> <abcd:Citations> <abcd:Citation abcd:language="EN"> <abcd:Text>Engels, Monika 2003 - (continuously updated) Catalogus novus et amplificatus speciminum et viventium algarum Desmidiacearum (New and extended catalogue of herbarium specimen and living material of Desmidiaceae in Germany).</abcd:Text> </abcd:Citation> </abcd:Citations> </abcd:IPRStatements> </abcd:Metadata> <abcd:Units> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-6239</abcd:UnitID> <abcd:UnitReferences> <abcd:UnitReference> <abcd:TitleCitation>Algae aquae dulcis exsiccatae praecique scandinavicae, quas adjectis algis marinis chlorophyllaceis et phycochromaceis distribuerunt., Stockholm 1877-1889, Vol. 1-26</abcd:TitleCitation> </abcd:UnitReference> </abcd:UnitReferences> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Mesotaeniaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Zygnematales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Ancylonema nordenskiöldii</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:Identifiers> <abcd:Identifier> <abcd:PersonName> <abcd:FullName>Nordstedt, O.</abcd:FullName> </abcd:PersonName> </abcd:Identifier> </abcd:Identifiers> <abcd:Date> <abcd:DateText>1878-07-18</abcd:DateText> </abcd:Date> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>Specimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:MultiMediaObjects> <abcd:MultiMediaObject> <abcd:FileURI>http://ww2.bgbm.org/herbarium/Externe_Bildsammlungen/AlgenEngels/L-Scans-B-Konv/6239.jpg</abcd:FileURI> </abcd:MultiMediaObject> </abcd:MultiMediaObjects> <abcd:Gathering> <abcd:DateTime>1878<abcd:DateText>1878</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Nordstedt, O.</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Brixdal</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>NO</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Nordfjord</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:AreaDetail language="DE">see label</abcd:AreaDetail> <abcd:Biotope> <abcd:Name language="EN">other</abcd:Name> <abcd:Text language="DE">see label</abcd:Text> </abcd:Biotope> <abcd:Notes language="EN">Europe</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: B 4000 08107; Specimen Location: B.</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-3397</abcd:UnitID> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Actinotaenium cucurbita (Brébisson) Teiling ex Ruzicka et Pouzar 1978</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 108, tab. 17: 7a-e as Cosmarium cucurbita; taxon citation: Teiling, E. 1954: Actinotaenium genus Desmidiacearum resuscitatum. - Bot. Notiser 100, p. 406, fig. 66 (invalidy pblished). Validation: Ruzicka, J. &amp; Pouzar, Z. 1978:Erwägungen über die Taxonomie und Nomenklatur der Gattung Actinotaenium TEIL. - Folia Geobot. Phytotax., Praha, 13: p.44</abcd:Notes> </abcd:Identification> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Cosmarium cucurbita (Bréb.) Ralfs 1848</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>false</abcd:PreferredFlag> <abcd:Identifiers> <abcd:Identifier> <abcd:PersonName> <abcd:FullName>Siegmund, W.</abcd:FullName> </abcd:PersonName> </abcd:Identifier> </abcd:Identifiers> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Date> <abcd:DateText>1872-07</abcd:DateText> </abcd:Date> <abcd:Notes>p. 108, tab. 17: 7a-e</abcd:Notes> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>Specimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:MultiMediaObjects> <abcd:MultiMediaObject> <abcd:FileURI>http://ww2.bgbm.org/herbarium/Externe_Bildsammlungen/AlgenEngels/L-Scans-HBG-Konv/3397.jpg</abcd:FileURI> </abcd:MultiMediaObject> </abcd:MultiMediaObjects> <abcd:Gathering> <abcd:DateTime>1872<abcd:DateText>1872</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Siegmund, W.</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Reichenberg</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>DE</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Sachsen</abcd:AreaName> </abcd:NamedArea> <abcd:NamedArea> <abcd:AreaName>Dresden</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:AreaDetail language="DE">bei Reichenberg, see label</abcd:AreaDetail> <abcd:Biotope> <abcd:Name language="EN">other</abcd:Name> <abcd:Text language="DE">see label</abcd:Text> </abcd:Biotope> <abcd:Notes language="EN">Europe</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: 8186; Specimen Location: HBG.</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-2483</abcd:UnitID> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Cosmarium turgidum Bréb. 1848</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>false</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 110, tab. 32: 8</abcd:Notes> </abcd:Identification> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Actinotaenium turgidum (Brébisson) Teiling 1954 ex Ruzicka et Pouzar 1978</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>as Cosmarium turgidum p. 110, tab. 32: 8; taxon citation: invalidly published as A. turgidum (Bréb.) Teiling, E. 1954: Actinotaenium genus Desmidiacearum resuscitatum. - Bot. Notiser 100 - p.395, fig.21; Validation by R?ži?ka &amp; Pouzar 1978:Erwägungen über die Taxonomie und Nomenklatur der Gattung Actinotaenium TEIL. - Folia Geobot. Phytotax., Praha, 13: 13 - 66 p.60</abcd:Notes> </abcd:Identification> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Pleurotaenium turgidum De Bary 1858</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>false</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>De Bary, A: Untersuchungen über die Familie der Conjugaten (Zygnemeen und Desmidieen)., Leipzig 1858</abcd:TitleCitation> <abcd:CitationDetail>p. 91</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 75, tab. 5: 31; invalidly published</abcd:Notes> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>Specimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:Gathering> <abcd:DateTime>0<abcd:DateText>0</abcd:DateText> </abcd:DateTime> <abcd:LocalityText language="DE">Neukirch</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>DE</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Sachsen</abcd:AreaName> </abcd:NamedArea> <abcd:NamedArea> <abcd:AreaName>Chemnitz</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:AreaDetail language="DE">Oberlausitz, Neukirch</abcd:AreaDetail> <abcd:Biotope> <abcd:Name language="EN" /> <abcd:Text language="DE" /> </abcd:Biotope> <abcd:Notes language="EN">Europe</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: 7474; Specimen Location: HBG. together with Closterium hybridum, Pl. truncatum</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-4603</abcd:UnitID> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Arthrodesmus octocornis var. major Roy &amp; Biss. 1894</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:Identifiers> <abcd:Identifier> <abcd:PersonName> <abcd:FullName>Richter, Paul</abcd:FullName> </abcd:PersonName> </abcd:Identifier> </abcd:Identifiers> <abcd:Notes>see label</abcd:Notes> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>Specimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:MultiMediaObjects> <abcd:MultiMediaObject> <abcd:FileURI>http://ww2.bgbm.org/herbarium/Externe_Bildsammlungen/AlgenEngels/L-Scans-HBG-Konv/4602.jpg</abcd:FileURI> </abcd:MultiMediaObject> </abcd:MultiMediaObjects> <abcd:Gathering> <abcd:DateTime>0<abcd:DateText>0</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Richter, Paul</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Leipzig</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>DE</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Sachsen</abcd:AreaName> </abcd:NamedArea> <abcd:NamedArea> <abcd:AreaName>Leipzig</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:AreaDetail language="DE" /> <abcd:Biotope> <abcd:Name language="EN" /> <abcd:Text language="DE" /> </abcd:Biotope> <abcd:Notes language="EN">Europe</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: 8773; Specimen Location: HBG.</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-6764</abcd:UnitID> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Actinotaenium curtum var. attenuatum (Bréb.) Teil. ex Ruzicka &amp; Pouzar 1978</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>as Cosmarium attenuatum p. 110, tab. 17: 9; taxon citation: Teiling, E., 1954: Actinotaenium genus Desmidiacearum resuscitatum. - Bot. Not., Lund, 1954, p. 391, fig. 2; invaldly published; Validation: Ruzicka, J., &amp; Pouzar, Z., 1978: Erwägungen über die Taxonomie und Nomenklatur der Gattung ActinotaeniumTEIL. - Folia geobot. Phytotax., Praha, 13: 33-66., p.47</abcd:Notes> </abcd:Identification> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Cosmarium attenuatum Ralfs 1848</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>false</abcd:PreferredFlag> <abcd:Identifiers> <abcd:Identifier> <abcd:PersonName> <abcd:FullName>Richter, Paul</abcd:FullName> </abcd:PersonName> </abcd:Identifier> </abcd:Identifiers> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 110, tab. 17: 9a-d</abcd:Notes> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>Specimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:Gathering> <abcd:DateTime>1898<abcd:DateText>1898</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Rogers,</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Ennepetal-Milspe</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>DE</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Nordrhein-Westfalen</abcd:AreaName> </abcd:NamedArea> <abcd:NamedArea> <abcd:AreaName>Hagen</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:AreaDetail language="DE" /> <abcd:Biotope> <abcd:Name language="EN" /> <abcd:Text language="DE">an einer Felsenwand</abcd:Text> </abcd:Biotope> <abcd:Notes language="EN">Europe</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: B 4000 08307; Specimen Location: B.</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-3698</abcd:UnitID> <abcd:UnitReferences> <abcd:UnitReference> <abcd:TitleCitation>Algae aquae dulcis exsiccatae praecique scandinavicae, quas adjectis algis marinis chlorophyllaceis et phycochromaceis distribuerunt., Lund 1896-1903, Vol. 27-34</abcd:TitleCitation> </abcd:UnitReference> </abcd:UnitReferences> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Actinotaenium palangula (Brébisson) Teiling 1954 ex Ruzicka et Pouzar 1978</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 212; original figure: Brébisson 1856: p.132, tab.1: 21; taxon citation: Ruzicka, J. &amp; Pouzar, Z., 1978: Erwägungen über die Taxonomie und Nomenklatur der Gattung Actinotaenium TEIL. - Folia Geobot. Phytotax., Praha, 13: p.46</abcd:Notes> </abcd:Identification> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Cosmarium palangula Brébisson in Ralfs 1848</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>false</abcd:PreferredFlag> <abcd:Identifiers> <abcd:Identifier> <abcd:PersonName> <abcd:FullName>Bohlin, Knut</abcd:FullName> </abcd:PersonName> </abcd:Identifier> </abcd:Identifiers> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Date> <abcd:DateText>1898-07-06</abcd:DateText> </abcd:Date> <abcd:Notes>p. 212; original figure: Brébisson 1856: p.132, tab.1: 21</abcd:Notes> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>Specimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:MultiMediaObjects> <abcd:MultiMediaObject> <abcd:FileURI>http://ww2.bgbm.org/herbarium/Externe_Bildsammlungen/AlgenEngels/L-Scans-HBG-Konv/3690-3703.jpg</abcd:FileURI> </abcd:MultiMediaObject> </abcd:MultiMediaObjects> <abcd:Gathering> <abcd:DateTime>1898<abcd:DateText>1898</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Bohlin, Knut</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Pico da Cruz</abcd:LocalityText> <abcd:AreaDetail language="DE" /> <abcd:Biotope> <abcd:Name language="EN" /> <abcd:Text language="DE" /> </abcd:Biotope> <abcd:Notes language="EN">Africa</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: 8328; Specimen Location: HBG. sample contains: Cosmarium ralfsii var. azoricum, Cosmarium palangula, Cosmarium pyramidatum, Cosmarium venustum, Euastru didelta f. scorbiculata, Staurastrum muticum var. depressum, Staurastrum scabrum</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-4569</abcd:UnitID> <abcd:UnitReferences> <abcd:UnitReference> <abcd:TitleCitation>Die Algen Europa's (Fortsetzung der Algen Sachsens, resp. Mittel-Europa's)., Dresden 1861 - 1879, Vol. Dec. 101 - 259</abcd:TitleCitation> </abcd:UnitReference> </abcd:UnitReferences> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Staurodesmus convergens (Ehrenberg ex Ralfs) Teiling 1967</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 118, tab. 20: 3a-d; taxon citation: Teiling, E., 1948: Staurodesmus, genus novum.Containing monospinous desmids. - Bot. Not., Lund, 1948: p. 57</abcd:Notes> </abcd:Identification> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Arthrodesmus convergens Ehrenberg ex Ralfs 1848</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>false</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 118, tab. 20: 3a-d</abcd:Notes> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>Specimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:Gathering> <abcd:DateTime>0<abcd:DateText>0</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Rostock, M.</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Dretschen</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>DE</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Sachsen</abcd:AreaName> </abcd:NamedArea> <abcd:NamedArea> <abcd:AreaName>Bautzen</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:AreaDetail language="DE" /> <abcd:Biotope> <abcd:Name language="EN">peat bog</abcd:Name> <abcd:Text language="DE" /> </abcd:Biotope> <abcd:Notes language="EN">Europe</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: 8751; Specimen Location: HBG.</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-7071</abcd:UnitID> <abcd:UnitReferences> <abcd:UnitReference> <abcd:TitleCitation>Kryptogamae exsiccatae editae a museo hist. natur. vindobonensi., Wien 1894 ff</abcd:TitleCitation> </abcd:UnitReference> </abcd:UnitReferences> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Arthrodesmus quadricaudatus</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:Identifiers> <abcd:Identifier> <abcd:PersonName> <abcd:FullName>Hansgirg, A.</abcd:FullName> </abcd:PersonName> </abcd:Identifier> </abcd:Identifiers> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>Specimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:Gathering> <abcd:DateTime>0<abcd:DateText>0</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Hansgirg, A.</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Bombay</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>IN</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Bombay</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:AreaDetail language="DE" /> <abcd:Biotope> <abcd:Name language="EN">other</abcd:Name> <abcd:Text language="DE">see label</abcd:Text> </abcd:Biotope> <abcd:Notes language="EN">Asia</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: B 4000 08385; Specimen Location: B. sample contains: Cosmarium nitidulum, Arthrodesmus quadricaudatus</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-6250</abcd:UnitID> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Staurodesmus incus (Brébisson ex Ralfs) Teiling 1967</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 118, tab. 20: 4 as Arthrodesmus incus; taxon citation: Teiling, E., 1967: The desmid genus Staurodesmus. A taxonomic study. Ark. f. Bot. II, 6(11), p. p. 511, tab. 5: 9</abcd:Notes> </abcd:Identification> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Arthrodesmus incus (Bréb.) Hassall ex Ralfs 1848</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>false</abcd:PreferredFlag> <abcd:Identifiers> <abcd:Identifier> <abcd:PersonName> <abcd:FullName>Borge, O.</abcd:FullName> </abcd:PersonName> </abcd:Identifier> </abcd:Identifiers> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Date> <abcd:DateText>1889-08-08</abcd:DateText> </abcd:Date> <abcd:Notes>p. 118, tab. 20: 4a-l</abcd:Notes> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>Specimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:Gathering> <abcd:DateTime>1889<abcd:DateText>1889</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Borge, O.</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Väddö</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>SE</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Upland</abcd:AreaName> </abcd:NamedArea> <abcd:NamedArea> <abcd:AreaName>Väddö</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:AreaDetail language="DE" /> <abcd:Biotope> <abcd:Name language="EN" /> <abcd:Text language="DE" /> </abcd:Biotope> <abcd:Notes language="EN">Europe</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: B 4000 32088; Specimen Location: B.</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-6897</abcd:UnitID> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Actinotaenium cucurbita (Brébisson) Teiling ex Ruzicka et Pouzar 1978</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 108, tab. 17: 7a-e as Cosmarium cucurbita; taxon citation: Teiling, E. 1954: Actinotaenium genus Desmidiacearum resuscitatum. - Bot. Notiser 100, p. 406, fig. 66 (invalidy pblished). Validation: Ruzicka, J. &amp; Pouzar, Z. 1978:Erwägungen über die Taxonomie und Nomenklatur der Gattung Actinotaenium TEIL. - Folia Geobot. Phytotax., Praha, 13: p.44</abcd:Notes> </abcd:Identification> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Cosmarium cucurbita (Bréb.) Ralfs 1848</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>false</abcd:PreferredFlag> <abcd:Identifiers> <abcd:Identifier> <abcd:PersonName> <abcd:FullName>Bulnheim, O.</abcd:FullName> </abcd:PersonName> </abcd:Identifier> </abcd:Identifiers> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Date> <abcd:DateText>1883-07</abcd:DateText> </abcd:Date> <abcd:Notes>p. 108, tab. 17: 7a-e</abcd:Notes> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>Specimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:MultiMediaObjects> <abcd:MultiMediaObject> <abcd:FileURI>http://ww2.bgbm.org/herbarium/Externe_Bildsammlungen/AlgenEngels/L-Scans-B-Konv/6897-6899.jpg</abcd:FileURI> </abcd:MultiMediaObject> </abcd:MultiMediaObjects> <abcd:Gathering> <abcd:DateTime>1883<abcd:DateText>1883</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Bulnheim, O.</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Leipzig</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>DE</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Sachsen</abcd:AreaName> </abcd:NamedArea> <abcd:NamedArea> <abcd:AreaName>Leipzig</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:AreaDetail language="DE">see label</abcd:AreaDetail> <abcd:Biotope> <abcd:Name language="EN" /> <abcd:Text language="DE" /> </abcd:Biotope> <abcd:Notes language="EN">Europe</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: B 4000 08358; Specimen Location: B. sample contains: Cosmarium cucurbita, Euastrum didelta, Closterium intermedium</abcd:Notes> </abcd:Unit> </abcd:Units> </abcd:DataSet> </abcd:DataSets> </biocase:content> <biocase:diagnostics> <biocase:diagnostic severity="INFO">Datasource wrapper AlgenEngels requested</biocase:diagnostic> <biocase:diagnostic severity="INFO">Reading PSF from C:\Workspace\bps2\config\datasources\AlgenEngels\provider_setup_file.xml</biocase:diagnostic> <biocase:diagnostic severity="DEBUG">PSF: PSF=C:\Workspace\bps2\config\datasources\AlgenEngels\provider_setup_file.xml, recLimit=0, loglevel=40, user=WebUser, database=AlgenEngels, dbIP=, dbms=odbc_tsql, encoding=latin_1, schemas={u'http://www.tdwg.org/schemas/abcd/2.06': &lt;biocase.wrapper.psf_handler.SupportedSchema instance at 0x00EB1E90&gt;}, tablegraph=GRAPH: graph: meta, determination-taxonNames, higherTaxon-taxonNames, images-specimen, namedAreas-collectingEvent, siteFeatures-collectingEvent, specimen-determination, specimen-collectingEvent, unitReference-specimen, +++ ALIAS2TABLE: {u'collectingEvent': u'abcd_CollectingEvent', u'taxonNames': u'abcd_TaxonName', u'higherTaxon': u'abcd_HigherTaxa', u'specimen': u'abcd_Object', u'siteFeatures': u'abcd_SiteFeatureTerms', u'meta': u'metadata', u'determination': u'abcd_Determination', u'images': u'abcd_Images', u'unitReference': u'abcd_UnitReference', u'namedAreas': u'abcd_NamedAreaNames'}</biocase:diagnostic> </biocase:diagnostics> </biocase:response> </pre> 06956a2fd3c335048f252cec52d96d602af29dee 0 41 95 2011-06-30T14:55:49Z JoergHoletschek 3 Created page with "This tutorial explains how to debug a BioCASe web service and a BioCASe installation in general. If you follow the instructions and don’t manage to get your service running, pl..." wikitext text/x-wiki This tutorial explains how to debug a BioCASe web service and a BioCASe installation in general. If you follow the instructions and don’t manage to get your service running, please contact the __BioCASe Team__. == Debugging a BioCASe Web Service == Debugging a BioCASe web service can be done with the manual Query Form. It allows you to load a BioCASe request (Search/Scan) from a template, edit the parameters of the request (e.g. the filter) and will show you the response document, which holds the requested ABCD records and useful debug information. == Opening the Query Form == There are several ways to open the Query Form. On the entry page of a datasource, <small>TEST and DEBUG</small> will open the Query Form: In the data source configuration tool, the link to the according query form can be always found in the link list at the top of the page: In the mapping editor, the button <small>Test mapping!</test> will open the Query Form in a separate tab: == Basic Usage == On top of the Query Form you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, flora being the name of the datasource. Make sure this is the correct access point of the web service you want to debug! (Bild) The usage of the Query Form is very straight forward: • Click on one of the links below the text box to load a template, for example for an ABCD2 Scan or ABCD2 Search; • Edit the request parameters; • Click on <small>Submit</small> to get the response document. The section <small>BioCASe protocol filter operators</small> lists the operators that can be used for creating complex filters that combine several filter criteria. More about creating filters can be found in the BioCASe Protocol documentation: http://www.biocase.org/products/protocols/index.shtml. Clicking on the <small>Show</small> link in the <small>Capabilities</small> section will show the Capabilities response in a frame of the Query Form. This is useful, since it shows the namespaces for the schemas supported by the web service (which is required in the request text) and XPaths of mapped ABCD elements (required for creating filters on these elements). By using Copy & Paste you can spare the trouble of entering these monstrous strings of characters. == Changing the Debug Level for a Data Source == Debug info can be set to one of four levels for a BioCASe data source: • <strong>Error</strong> is the highest level will only output errors that prevent the web server from answering a request. • <strong>Warning</strong> will output errors that keep BioCASe from answering a request and warnings that don’t prevent the fulfilling of the request, but should be analysed carefully. For example, they might tell you that records have been dropped because mandatory ABCD elements have not been mapped or were empty for certain records. • <strong>Info</strong> is the default and will add some more information about how the Provider Software processes the request. • <strong>Debug</strong> is the most detailed level and will produce very verbose debug output. It adds very comprehensive information about how the ABCD documents are constructed and is suitable only for experienced users. For most cases, keeping the default Info level should be sufficient for testing and debugging a web service. For production use, you should set it to Error later. You can do this in the configuration of a datasource under Settings: == Analyzing the Response Document == Upon clicking <small>Submit</small>, the Query Form will send the request to the web service and wait for the response. It might take some time – up to minutes – until the response document shows up. Amongst others, this depends on • the complexity of the query (especially the filters), • the size of the dataset returned, • the debug level set (“debug” results in very exhaustive debug output), • the performance of the servers running BioCASe and the database, • the network bandwidth. (bild) The response document consists of three sections. A full response document can be found here. === Header == This section contains information about the BioCASe installation – operating system, DBMS used, Python/BioCASe version etc. It is intended for a consumer application and won’t be very helpful for debugging: <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> === Content === This section makes up the payload of the response document, depending on the type of request: Capabilities: Lists all schemas supported by the web service. For each schema, all mapped concepts (identified by their XPaths) are listed with data type and searchable flag. Scan: The document will simply list all distinct values for the concept specified in the request. When done for the scientific name, it would look similar to this: <biocase:content recordCount="1664" recordDropped="0" recordStart="0" totalSearchHits="1664"><biocase:scan><biocase:value>Achillea atrata L.</biocase:value><biocase:value>Achillea millefolium L.</biocase:value><biocase:value>Achillea nobilis L. ssp. nobilis</biocase:value><biocase:value>Achillea ptarmica L.</biocase:value><biocase:value>Acinos alpinus (L.) Moench</biocase:value><biocase:value>Acinos arvensis (Lam.) Dandy</biocase:value><biocase:value>Aconitum degenii ssp. paniculatum (Arcang.) Mucher</biocase:value><biocase:value>Aconitum lycoctonum L. ssp. lycoctonum</biocase:value><biocase:value>Aconitum napellus L. s.l.</biocase:value> Search: For this type of request, the content consists of the ABCD dataset(s). See the ABCD2 example. === Debug Output === This section is the most interesting part for debugging. If debug level is set to Info (which is recommended for debugging), it will document the processing of the request by the web service. The output is easy to understand, for example it will contain the SQL statements sent to the database. For a successful search request it could look similar to this: == Sample Errors and Warnings in the Debug Output == <biocase:diagnostic severity="ERROR"> No DB connection could be established. Please verify DB state and connection parameters in your provider_setup_file.xml. </biocase:diagnostic> Well… just what it says. The Provider Software could not connect to the database to be published. Check the database connection settings (server IP, database name, user, and password) and make sure you’ve installed the Python package for the DBMS you’re using (see the Libs test page in Utilities). <biocase:diagnostic severity="INFO">Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06</biocase:diagnostic><biocase:diagnostic severity="INFO">Load CMFile 'C:\Workspace\bps2\config\datasources\flora\cmf_ABCD_2.06.xml'</biocase:diagnostic><biocase:diagnostic severity="ERROR">The CMF DB mapping file http://www.tdwg.org/schemas/abcd/2.06 was corrupt or could not be interpreted.</biocase:diagnostic> The requested schema (identified by the namespace http://www.tdwg.org/schemas/abcd/2.06) is not supported by that service. Create a mapping and map elements. <biocase:diagnostic severity="ERROR">The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts</biocase:diagnostic> Obvious - happens if you try to use a concept that is not mapped in a scan request or a filter. <biocase:diagnostic severity="ERROR">A SQL statement produced an error: SELECT unit.id, metadata.supplier_organisation, metadata.supplier_email, …, unit.febnr, unit.nameautor, unit.url_gbild, unit.datum1, unit.sammler, unit.fundort, unit.hoehe, metadata.id, unit.id FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) ORDER BY unit.id That could have a number of reasons: One of the table/column names that were set up in the configuration does not exist (because it was renamed or removed) or the credentials used by the BPS do not have sufficient privileges. Simply copy the SQL statement and execute it manually on the database with a regular database client that will show you the detailed error message returned by the DBMS. <biocase:diagnostic severity="WARNING">The mandatory child element SourceInstitutionID could not be created [0 existing, but 1 required]. The element /DataSets/DataSet/Units/Unit was also dropped.</biocase:diagnostic><biocase:diagnostic severity="WARNING">The mandatory child element Units could not be created [0 existing, but 1 required]. The element /DataSets/DataSet was also dropped.</biocase:diagnostic> One problem caused the next one in this sequence of warnings: The mandatory ABCD element was either not mapped or empty; therefore the respective record (“unit”) was dropped. Because this happened for all units in the datasets, all units were dropped, resulting in an empty Units sub tree. The second warning tells you what happened then: Because the Units element is mandatory for the Dataset element, the whole dataset was dropped. As a result, the response ABCD document was empty. == Debugging a BioCASe Installation == In case the debug output of the web service doesn’t help you to find the problem, or if the BPS behaves not as intended without displaying any error messages, you should have a look at the debug logs written by the BPS. To turn on the Debug logs, go to the System Administration page and set Debug logs to True: Bild Redo the step that didn’t work and open the log folder of your BioCASe installation. There are several global log files (pywrapper_request.log, pywrapper_error.log, webapp_debug.log, webapp_error.log) and a debug files per data source (e.g. debug_flora.log). Open these files and see if they can give you a clue to the problem. If you still can’t get it running, contact the __BioCASe Team__. 6e7087a1956d92c371b8806c1c2bbe03dbc91df4 96 95 2011-06-30T14:57:35Z JoergHoletschek 3 wikitext text/x-wiki This tutorial explains how to debug a BioCASe web service and a BioCASe installation in general. If you follow the instructions and don’t manage to get your service running, please contact the __BioCASe Team__. == Debugging a BioCASe Web Service == Debugging a BioCASe web service can be done with the manual Query Form. It allows you to load a BioCASe request (Search/Scan) from a template, edit the parameters of the request (e.g. the filter) and will show you the response document, which holds the requested ABCD records and useful debug information. === Opening the Query Form === There are several ways to open the Query Form. On the entry page of a datasource, <small>TEST and DEBUG</small> will open the Query Form: In the data source configuration tool, the link to the according query form can be always found in the link list at the top of the page: In the mapping editor, the button <small>Test mapping!</test> will open the Query Form in a separate tab: === Basic Usage === On top of the Query Form you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, flora being the name of the datasource. Make sure this is the correct access point of the web service you want to debug! (Bild) The usage of the Query Form is very straight forward: • Click on one of the links below the text box to load a template, for example for an ABCD2 Scan or ABCD2 Search; • Edit the request parameters; • Click on <small>Submit</small> to get the response document. The section <small>BioCASe protocol filter operators</small> lists the operators that can be used for creating complex filters that combine several filter criteria. More about creating filters can be found in the BioCASe Protocol documentation: http://www.biocase.org/products/protocols/index.shtml. Clicking on the <small>Show</small> link in the <small>Capabilities</small> section will show the Capabilities response in a frame of the Query Form. This is useful, since it shows the namespaces for the schemas supported by the web service (which is required in the request text) and XPaths of mapped ABCD elements (required for creating filters on these elements). By using Copy & Paste you can spare the trouble of entering these monstrous strings of characters. === Changing the Debug Level for a Data Source === Debug info can be set to one of four levels for a BioCASe data source: • <strong>Error</strong> is the highest level will only output errors that prevent the web server from answering a request. • <strong>Warning</strong> will output errors that keep BioCASe from answering a request and warnings that don’t prevent the fulfilling of the request, but should be analysed carefully. For example, they might tell you that records have been dropped because mandatory ABCD elements have not been mapped or were empty for certain records. • <strong>Info</strong> is the default and will add some more information about how the Provider Software processes the request. • <strong>Debug</strong> is the most detailed level and will produce very verbose debug output. It adds very comprehensive information about how the ABCD documents are constructed and is suitable only for experienced users. For most cases, keeping the default Info level should be sufficient for testing and debugging a web service. For production use, you should set it to Error later. You can do this in the configuration of a datasource under Settings: == Analyzing the Response Document == Upon clicking <small>Submit</small>, the Query Form will send the request to the web service and wait for the response. It might take some time – up to minutes – until the response document shows up. Amongst others, this depends on • the complexity of the query (especially the filters), • the size of the dataset returned, • the debug level set (“debug” results in very exhaustive debug output), • the performance of the servers running BioCASe and the database, • the network bandwidth. (bild) The response document consists of three sections. A full response document can be found here. === Header === This section contains information about the BioCASe installation – operating system, DBMS used, Python/BioCASe version etc. It is intended for a consumer application and won’t be very helpful for debugging: <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> === Content === This section makes up the payload of the response document, depending on the type of request: Capabilities: Lists all schemas supported by the web service. For each schema, all mapped concepts (identified by their XPaths) are listed with data type and searchable flag. Scan: The document will simply list all distinct values for the concept specified in the request. When done for the scientific name, it would look similar to this: <biocase:content recordCount="1664" recordDropped="0" recordStart="0" totalSearchHits="1664"><biocase:scan><biocase:value>Achillea atrata L.</biocase:value><biocase:value>Achillea millefolium L.</biocase:value><biocase:value>Achillea nobilis L. ssp. nobilis</biocase:value><biocase:value>Achillea ptarmica L.</biocase:value><biocase:value>Acinos alpinus (L.) Moench</biocase:value><biocase:value>Acinos arvensis (Lam.) Dandy</biocase:value><biocase:value>Aconitum degenii ssp. paniculatum (Arcang.) Mucher</biocase:value><biocase:value>Aconitum lycoctonum L. ssp. lycoctonum</biocase:value><biocase:value>Aconitum napellus L. s.l.</biocase:value> Search: For this type of request, the content consists of the ABCD dataset(s). See the ABCD2 example. === Debug Output === This section is the most interesting part for debugging. If debug level is set to Info (which is recommended for debugging), it will document the processing of the request by the web service. The output is easy to understand, for example it will contain the SQL statements sent to the database. For a successful search request it could look similar to this: == Sample Errors and Warnings in the Debug Output == <biocase:diagnostic severity="ERROR"> No DB connection could be established. Please verify DB state and connection parameters in your provider_setup_file.xml. </biocase:diagnostic> Well… just what it says. The Provider Software could not connect to the database to be published. Check the database connection settings (server IP, database name, user, and password) and make sure you’ve installed the Python package for the DBMS you’re using (see the Libs test page in Utilities). <biocase:diagnostic severity="INFO">Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06</biocase:diagnostic><biocase:diagnostic severity="INFO">Load CMFile 'C:\Workspace\bps2\config\datasources\flora\cmf_ABCD_2.06.xml'</biocase:diagnostic><biocase:diagnostic severity="ERROR">The CMF DB mapping file http://www.tdwg.org/schemas/abcd/2.06 was corrupt or could not be interpreted.</biocase:diagnostic> The requested schema (identified by the namespace http://www.tdwg.org/schemas/abcd/2.06) is not supported by that service. Create a mapping and map elements. <biocase:diagnostic severity="ERROR">The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts</biocase:diagnostic> Obvious - happens if you try to use a concept that is not mapped in a scan request or a filter. <biocase:diagnostic severity="ERROR">A SQL statement produced an error: SELECT unit.id, metadata.supplier_organisation, metadata.supplier_email, …, unit.febnr, unit.nameautor, unit.url_gbild, unit.datum1, unit.sammler, unit.fundort, unit.hoehe, metadata.id, unit.id FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) ORDER BY unit.id That could have a number of reasons: One of the table/column names that were set up in the configuration does not exist (because it was renamed or removed) or the credentials used by the BPS do not have sufficient privileges. Simply copy the SQL statement and execute it manually on the database with a regular database client that will show you the detailed error message returned by the DBMS. <biocase:diagnostic severity="WARNING">The mandatory child element SourceInstitutionID could not be created [0 existing, but 1 required]. The element /DataSets/DataSet/Units/Unit was also dropped.</biocase:diagnostic><biocase:diagnostic severity="WARNING">The mandatory child element Units could not be created [0 existing, but 1 required]. The element /DataSets/DataSet was also dropped.</biocase:diagnostic> One problem caused the next one in this sequence of warnings: The mandatory ABCD element was either not mapped or empty; therefore the respective record (“unit”) was dropped. Because this happened for all units in the datasets, all units were dropped, resulting in an empty Units sub tree. The second warning tells you what happened then: Because the Units element is mandatory for the Dataset element, the whole dataset was dropped. As a result, the response ABCD document was empty. == Debugging a BioCASe Installation == In case the debug output of the web service doesn’t help you to find the problem, or if the BPS behaves not as intended without displaying any error messages, you should have a look at the debug logs written by the BPS. To turn on the Debug logs, go to the System Administration page and set Debug logs to True: Bild Redo the step that didn’t work and open the log folder of your BioCASe installation. There are several global log files (pywrapper_request.log, pywrapper_error.log, webapp_debug.log, webapp_error.log) and a debug files per data source (e.g. debug_flora.log). Open these files and see if they can give you a clue to the problem. If you still can’t get it running, contact the __BioCASe Team__. 567c72ed94af7aecdbff4a5ff44056c655d95333 97 96 2011-06-30T15:01:30Z JoergHoletschek 3 /* Debugging a BioCASe Web Service */ wikitext text/x-wiki This tutorial explains how to debug a BioCASe web service and a BioCASe installation in general. If you follow the instructions and don’t manage to get your service running, please contact the __BioCASe Team__. == Debugging a BioCASe Web Service == Debugging a BioCASe web service can be done with the manual Query Form. It allows you to load a BioCASe request (Search/Scan) from a template, edit the parameters of the request (e.g. the filter) and will show you the response document, which holds the requested ABCD records and useful debug information. === Opening the Query Form === There are several ways to open the Query Form. On the entry page of a datasource, <small>TEST and DEBUG</small> will open the Query Form: [[File:debugDatasourceEntryPage.png]] In the data source configuration tool, the link to the according query form can be always found in the link list at the top of the page: [[File:debugDatasourceLinklist.png]] In the mapping editor, the button <small>Test mapping!</small> will open the Query Form in a separate tab: [[File:abcdMappingEditorEmptyTop.png]] === Basic Usage === On top of the Query Form you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, flora being the name of the datasource. Make sure this is the correct access point of the web service you want to debug! (Bild) The usage of the Query Form is very straight forward: • Click on one of the links below the text box to load a template, for example for an ABCD2 Scan or ABCD2 Search; • Edit the request parameters; • Click on <small>Submit</small> to get the response document. The section <small>BioCASe protocol filter operators</small> lists the operators that can be used for creating complex filters that combine several filter criteria. More about creating filters can be found in the BioCASe Protocol documentation: http://www.biocase.org/products/protocols/index.shtml. Clicking on the <small>Show</small> link in the <small>Capabilities</small> section will show the Capabilities response in a frame of the Query Form. This is useful, since it shows the namespaces for the schemas supported by the web service (which is required in the request text) and XPaths of mapped ABCD elements (required for creating filters on these elements). By using Copy & Paste you can spare the trouble of entering these monstrous strings of characters. === Changing the Debug Level for a Data Source === Debug info can be set to one of four levels for a BioCASe data source: • <strong>Error</strong> is the highest level will only output errors that prevent the web server from answering a request. • <strong>Warning</strong> will output errors that keep BioCASe from answering a request and warnings that don’t prevent the fulfilling of the request, but should be analysed carefully. For example, they might tell you that records have been dropped because mandatory ABCD elements have not been mapped or were empty for certain records. • <strong>Info</strong> is the default and will add some more information about how the Provider Software processes the request. • <strong>Debug</strong> is the most detailed level and will produce very verbose debug output. It adds very comprehensive information about how the ABCD documents are constructed and is suitable only for experienced users. For most cases, keeping the default Info level should be sufficient for testing and debugging a web service. For production use, you should set it to Error later. You can do this in the configuration of a datasource under Settings: == Analyzing the Response Document == Upon clicking <small>Submit</small>, the Query Form will send the request to the web service and wait for the response. It might take some time – up to minutes – until the response document shows up. Amongst others, this depends on • the complexity of the query (especially the filters), • the size of the dataset returned, • the debug level set (“debug” results in very exhaustive debug output), • the performance of the servers running BioCASe and the database, • the network bandwidth. (bild) The response document consists of three sections. A full response document can be found here. === Header === This section contains information about the BioCASe installation – operating system, DBMS used, Python/BioCASe version etc. It is intended for a consumer application and won’t be very helpful for debugging: <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> === Content === This section makes up the payload of the response document, depending on the type of request: Capabilities: Lists all schemas supported by the web service. For each schema, all mapped concepts (identified by their XPaths) are listed with data type and searchable flag. Scan: The document will simply list all distinct values for the concept specified in the request. When done for the scientific name, it would look similar to this: <biocase:content recordCount="1664" recordDropped="0" recordStart="0" totalSearchHits="1664"><biocase:scan><biocase:value>Achillea atrata L.</biocase:value><biocase:value>Achillea millefolium L.</biocase:value><biocase:value>Achillea nobilis L. ssp. nobilis</biocase:value><biocase:value>Achillea ptarmica L.</biocase:value><biocase:value>Acinos alpinus (L.) Moench</biocase:value><biocase:value>Acinos arvensis (Lam.) Dandy</biocase:value><biocase:value>Aconitum degenii ssp. paniculatum (Arcang.) Mucher</biocase:value><biocase:value>Aconitum lycoctonum L. ssp. lycoctonum</biocase:value><biocase:value>Aconitum napellus L. s.l.</biocase:value> Search: For this type of request, the content consists of the ABCD dataset(s). See the ABCD2 example. === Debug Output === This section is the most interesting part for debugging. If debug level is set to Info (which is recommended for debugging), it will document the processing of the request by the web service. The output is easy to understand, for example it will contain the SQL statements sent to the database. For a successful search request it could look similar to this: == Sample Errors and Warnings in the Debug Output == <biocase:diagnostic severity="ERROR"> No DB connection could be established. Please verify DB state and connection parameters in your provider_setup_file.xml. </biocase:diagnostic> Well… just what it says. The Provider Software could not connect to the database to be published. Check the database connection settings (server IP, database name, user, and password) and make sure you’ve installed the Python package for the DBMS you’re using (see the Libs test page in Utilities). <biocase:diagnostic severity="INFO">Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06</biocase:diagnostic><biocase:diagnostic severity="INFO">Load CMFile 'C:\Workspace\bps2\config\datasources\flora\cmf_ABCD_2.06.xml'</biocase:diagnostic><biocase:diagnostic severity="ERROR">The CMF DB mapping file http://www.tdwg.org/schemas/abcd/2.06 was corrupt or could not be interpreted.</biocase:diagnostic> The requested schema (identified by the namespace http://www.tdwg.org/schemas/abcd/2.06) is not supported by that service. Create a mapping and map elements. <biocase:diagnostic severity="ERROR">The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts</biocase:diagnostic> Obvious - happens if you try to use a concept that is not mapped in a scan request or a filter. <biocase:diagnostic severity="ERROR">A SQL statement produced an error: SELECT unit.id, metadata.supplier_organisation, metadata.supplier_email, …, unit.febnr, unit.nameautor, unit.url_gbild, unit.datum1, unit.sammler, unit.fundort, unit.hoehe, metadata.id, unit.id FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) ORDER BY unit.id That could have a number of reasons: One of the table/column names that were set up in the configuration does not exist (because it was renamed or removed) or the credentials used by the BPS do not have sufficient privileges. Simply copy the SQL statement and execute it manually on the database with a regular database client that will show you the detailed error message returned by the DBMS. <biocase:diagnostic severity="WARNING">The mandatory child element SourceInstitutionID could not be created [0 existing, but 1 required]. The element /DataSets/DataSet/Units/Unit was also dropped.</biocase:diagnostic><biocase:diagnostic severity="WARNING">The mandatory child element Units could not be created [0 existing, but 1 required]. The element /DataSets/DataSet was also dropped.</biocase:diagnostic> One problem caused the next one in this sequence of warnings: The mandatory ABCD element was either not mapped or empty; therefore the respective record (“unit”) was dropped. Because this happened for all units in the datasets, all units were dropped, resulting in an empty Units sub tree. The second warning tells you what happened then: Because the Units element is mandatory for the Dataset element, the whole dataset was dropped. As a result, the response ABCD document was empty. == Debugging a BioCASe Installation == In case the debug output of the web service doesn’t help you to find the problem, or if the BPS behaves not as intended without displaying any error messages, you should have a look at the debug logs written by the BPS. To turn on the Debug logs, go to the System Administration page and set Debug logs to True: Bild Redo the step that didn’t work and open the log folder of your BioCASe installation. There are several global log files (pywrapper_request.log, pywrapper_error.log, webapp_debug.log, webapp_error.log) and a debug files per data source (e.g. debug_flora.log). Open these files and see if they can give you a clue to the problem. If you still can’t get it running, contact the __BioCASe Team__. a6f421ad76852dea433db9a62f9bb3250ce87cb6 6 42 98 2011-06-30T15:03:19Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 43 99 2011-06-30T15:03:31Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 44 100 2011-06-30T15:03:46Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 0 41 101 97 2011-06-30T15:05:05Z JoergHoletschek 3 /* Opening the Query Form */ wikitext text/x-wiki This tutorial explains how to debug a BioCASe web service and a BioCASe installation in general. If you follow the instructions and don’t manage to get your service running, please contact the __BioCASe Team__. == Debugging a BioCASe Web Service == Debugging a BioCASe web service can be done with the manual Query Form. It allows you to load a BioCASe request (Search/Scan) from a template, edit the parameters of the request (e.g. the filter) and will show you the response document, which holds the requested ABCD records and useful debug information. === Opening the Query Form === There are several ways to open the Query Form. On the entry page of a datasource, <small>TEST and DEBUG</small> will open the Query Form: [[File:debugDatasourceEntryPage.png]] In the data source configuration tool, the link to the according query form can be always found in the link list at the top of the page. Moreover, in the mapping editor, the button <small>Test mapping!</small> will open the Query Form in a separate tab: [[File:abcdMappingEditorEmptyTop.png]] === Basic Usage === On top of the Query Form you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, flora being the name of the datasource. Make sure this is the correct access point of the web service you want to debug! (Bild) The usage of the Query Form is very straight forward: • Click on one of the links below the text box to load a template, for example for an ABCD2 Scan or ABCD2 Search; • Edit the request parameters; • Click on <small>Submit</small> to get the response document. The section <small>BioCASe protocol filter operators</small> lists the operators that can be used for creating complex filters that combine several filter criteria. More about creating filters can be found in the BioCASe Protocol documentation: http://www.biocase.org/products/protocols/index.shtml. Clicking on the <small>Show</small> link in the <small>Capabilities</small> section will show the Capabilities response in a frame of the Query Form. This is useful, since it shows the namespaces for the schemas supported by the web service (which is required in the request text) and XPaths of mapped ABCD elements (required for creating filters on these elements). By using Copy & Paste you can spare the trouble of entering these monstrous strings of characters. === Changing the Debug Level for a Data Source === Debug info can be set to one of four levels for a BioCASe data source: • <strong>Error</strong> is the highest level will only output errors that prevent the web server from answering a request. • <strong>Warning</strong> will output errors that keep BioCASe from answering a request and warnings that don’t prevent the fulfilling of the request, but should be analysed carefully. For example, they might tell you that records have been dropped because mandatory ABCD elements have not been mapped or were empty for certain records. • <strong>Info</strong> is the default and will add some more information about how the Provider Software processes the request. • <strong>Debug</strong> is the most detailed level and will produce very verbose debug output. It adds very comprehensive information about how the ABCD documents are constructed and is suitable only for experienced users. For most cases, keeping the default Info level should be sufficient for testing and debugging a web service. For production use, you should set it to Error later. You can do this in the configuration of a datasource under Settings: == Analyzing the Response Document == Upon clicking <small>Submit</small>, the Query Form will send the request to the web service and wait for the response. It might take some time – up to minutes – until the response document shows up. Amongst others, this depends on • the complexity of the query (especially the filters), • the size of the dataset returned, • the debug level set (“debug” results in very exhaustive debug output), • the performance of the servers running BioCASe and the database, • the network bandwidth. (bild) The response document consists of three sections. A full response document can be found here. === Header === This section contains information about the BioCASe installation – operating system, DBMS used, Python/BioCASe version etc. It is intended for a consumer application and won’t be very helpful for debugging: <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> === Content === This section makes up the payload of the response document, depending on the type of request: Capabilities: Lists all schemas supported by the web service. For each schema, all mapped concepts (identified by their XPaths) are listed with data type and searchable flag. Scan: The document will simply list all distinct values for the concept specified in the request. When done for the scientific name, it would look similar to this: <biocase:content recordCount="1664" recordDropped="0" recordStart="0" totalSearchHits="1664"><biocase:scan><biocase:value>Achillea atrata L.</biocase:value><biocase:value>Achillea millefolium L.</biocase:value><biocase:value>Achillea nobilis L. ssp. nobilis</biocase:value><biocase:value>Achillea ptarmica L.</biocase:value><biocase:value>Acinos alpinus (L.) Moench</biocase:value><biocase:value>Acinos arvensis (Lam.) Dandy</biocase:value><biocase:value>Aconitum degenii ssp. paniculatum (Arcang.) Mucher</biocase:value><biocase:value>Aconitum lycoctonum L. ssp. lycoctonum</biocase:value><biocase:value>Aconitum napellus L. s.l.</biocase:value> Search: For this type of request, the content consists of the ABCD dataset(s). See the ABCD2 example. === Debug Output === This section is the most interesting part for debugging. If debug level is set to Info (which is recommended for debugging), it will document the processing of the request by the web service. The output is easy to understand, for example it will contain the SQL statements sent to the database. For a successful search request it could look similar to this: == Sample Errors and Warnings in the Debug Output == <biocase:diagnostic severity="ERROR"> No DB connection could be established. Please verify DB state and connection parameters in your provider_setup_file.xml. </biocase:diagnostic> Well… just what it says. The Provider Software could not connect to the database to be published. Check the database connection settings (server IP, database name, user, and password) and make sure you’ve installed the Python package for the DBMS you’re using (see the Libs test page in Utilities). <biocase:diagnostic severity="INFO">Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06</biocase:diagnostic><biocase:diagnostic severity="INFO">Load CMFile 'C:\Workspace\bps2\config\datasources\flora\cmf_ABCD_2.06.xml'</biocase:diagnostic><biocase:diagnostic severity="ERROR">The CMF DB mapping file http://www.tdwg.org/schemas/abcd/2.06 was corrupt or could not be interpreted.</biocase:diagnostic> The requested schema (identified by the namespace http://www.tdwg.org/schemas/abcd/2.06) is not supported by that service. Create a mapping and map elements. <biocase:diagnostic severity="ERROR">The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts</biocase:diagnostic> Obvious - happens if you try to use a concept that is not mapped in a scan request or a filter. <biocase:diagnostic severity="ERROR">A SQL statement produced an error: SELECT unit.id, metadata.supplier_organisation, metadata.supplier_email, …, unit.febnr, unit.nameautor, unit.url_gbild, unit.datum1, unit.sammler, unit.fundort, unit.hoehe, metadata.id, unit.id FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) ORDER BY unit.id That could have a number of reasons: One of the table/column names that were set up in the configuration does not exist (because it was renamed or removed) or the credentials used by the BPS do not have sufficient privileges. Simply copy the SQL statement and execute it manually on the database with a regular database client that will show you the detailed error message returned by the DBMS. <biocase:diagnostic severity="WARNING">The mandatory child element SourceInstitutionID could not be created [0 existing, but 1 required]. The element /DataSets/DataSet/Units/Unit was also dropped.</biocase:diagnostic><biocase:diagnostic severity="WARNING">The mandatory child element Units could not be created [0 existing, but 1 required]. The element /DataSets/DataSet was also dropped.</biocase:diagnostic> One problem caused the next one in this sequence of warnings: The mandatory ABCD element was either not mapped or empty; therefore the respective record (“unit”) was dropped. Because this happened for all units in the datasets, all units were dropped, resulting in an empty Units sub tree. The second warning tells you what happened then: Because the Units element is mandatory for the Dataset element, the whole dataset was dropped. As a result, the response ABCD document was empty. == Debugging a BioCASe Installation == In case the debug output of the web service doesn’t help you to find the problem, or if the BPS behaves not as intended without displaying any error messages, you should have a look at the debug logs written by the BPS. To turn on the Debug logs, go to the System Administration page and set Debug logs to True: Bild Redo the step that didn’t work and open the log folder of your BioCASe installation. There are several global log files (pywrapper_request.log, pywrapper_error.log, webapp_debug.log, webapp_error.log) and a debug files per data source (e.g. debug_flora.log). Open these files and see if they can give you a clue to the problem. If you still can’t get it running, contact the __BioCASe Team__. 58d90060e253a5b649471e59fd1577aba7249836 102 101 2011-06-30T15:07:06Z JoergHoletschek 3 /* Basic Usage */ wikitext text/x-wiki This tutorial explains how to debug a BioCASe web service and a BioCASe installation in general. If you follow the instructions and don’t manage to get your service running, please contact the __BioCASe Team__. == Debugging a BioCASe Web Service == Debugging a BioCASe web service can be done with the manual Query Form. It allows you to load a BioCASe request (Search/Scan) from a template, edit the parameters of the request (e.g. the filter) and will show you the response document, which holds the requested ABCD records and useful debug information. === Opening the Query Form === There are several ways to open the Query Form. On the entry page of a datasource, <small>TEST and DEBUG</small> will open the Query Form: [[File:debugDatasourceEntryPage.png]] In the data source configuration tool, the link to the according query form can be always found in the link list at the top of the page. Moreover, in the mapping editor, the button <small>Test mapping!</small> will open the Query Form in a separate tab: [[File:abcdMappingEditorEmptyTop.png]] === Basic Usage === On top of the Query Form you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ''flora'' being the name of the datasource. Make sure this is the correct access point of the web service you want to debug! [[File:debugQueryformFull.png]] The usage of the Query Form is very straight forward: * Click on one of the links below the text box to load a template, for example for an ABCD2 Scan or ABCD2 Search; * Edit the request parameters; * Click on <small>Submit</small> to get the response document. The section <small>BioCASe protocol filter operators</small> lists the operators that can be used for creating complex filters that combine several filter criteria. More about creating filters can be found in the BioCASe Protocol documentation at http://www.biocase.org/products/protocols/index.shtml. Clicking on the <small>Show</small> link in the <small>Capabilities</small> section will show the Capabilities response in a frame of the Query Form. This is useful, since it shows the namespaces for the schemas supported by the web service (which is required in the request text) and XPaths of mapped ABCD elements (required for creating filters on these elements). By using Copy & Paste you can spare the trouble of entering these monstrous strings of characters. === Changing the Debug Level for a Data Source === Debug info can be set to one of four levels for a BioCASe data source: • <strong>Error</strong> is the highest level will only output errors that prevent the web server from answering a request. • <strong>Warning</strong> will output errors that keep BioCASe from answering a request and warnings that don’t prevent the fulfilling of the request, but should be analysed carefully. For example, they might tell you that records have been dropped because mandatory ABCD elements have not been mapped or were empty for certain records. • <strong>Info</strong> is the default and will add some more information about how the Provider Software processes the request. • <strong>Debug</strong> is the most detailed level and will produce very verbose debug output. It adds very comprehensive information about how the ABCD documents are constructed and is suitable only for experienced users. For most cases, keeping the default Info level should be sufficient for testing and debugging a web service. For production use, you should set it to Error later. You can do this in the configuration of a datasource under Settings: == Analyzing the Response Document == Upon clicking <small>Submit</small>, the Query Form will send the request to the web service and wait for the response. It might take some time – up to minutes – until the response document shows up. Amongst others, this depends on • the complexity of the query (especially the filters), • the size of the dataset returned, • the debug level set (“debug” results in very exhaustive debug output), • the performance of the servers running BioCASe and the database, • the network bandwidth. (bild) The response document consists of three sections. A full response document can be found here. === Header === This section contains information about the BioCASe installation – operating system, DBMS used, Python/BioCASe version etc. It is intended for a consumer application and won’t be very helpful for debugging: <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> === Content === This section makes up the payload of the response document, depending on the type of request: Capabilities: Lists all schemas supported by the web service. For each schema, all mapped concepts (identified by their XPaths) are listed with data type and searchable flag. Scan: The document will simply list all distinct values for the concept specified in the request. When done for the scientific name, it would look similar to this: <biocase:content recordCount="1664" recordDropped="0" recordStart="0" totalSearchHits="1664"><biocase:scan><biocase:value>Achillea atrata L.</biocase:value><biocase:value>Achillea millefolium L.</biocase:value><biocase:value>Achillea nobilis L. ssp. nobilis</biocase:value><biocase:value>Achillea ptarmica L.</biocase:value><biocase:value>Acinos alpinus (L.) Moench</biocase:value><biocase:value>Acinos arvensis (Lam.) Dandy</biocase:value><biocase:value>Aconitum degenii ssp. paniculatum (Arcang.) Mucher</biocase:value><biocase:value>Aconitum lycoctonum L. ssp. lycoctonum</biocase:value><biocase:value>Aconitum napellus L. s.l.</biocase:value> Search: For this type of request, the content consists of the ABCD dataset(s). See the ABCD2 example. === Debug Output === This section is the most interesting part for debugging. If debug level is set to Info (which is recommended for debugging), it will document the processing of the request by the web service. The output is easy to understand, for example it will contain the SQL statements sent to the database. For a successful search request it could look similar to this: == Sample Errors and Warnings in the Debug Output == <biocase:diagnostic severity="ERROR"> No DB connection could be established. Please verify DB state and connection parameters in your provider_setup_file.xml. </biocase:diagnostic> Well… just what it says. The Provider Software could not connect to the database to be published. Check the database connection settings (server IP, database name, user, and password) and make sure you’ve installed the Python package for the DBMS you’re using (see the Libs test page in Utilities). <biocase:diagnostic severity="INFO">Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06</biocase:diagnostic><biocase:diagnostic severity="INFO">Load CMFile 'C:\Workspace\bps2\config\datasources\flora\cmf_ABCD_2.06.xml'</biocase:diagnostic><biocase:diagnostic severity="ERROR">The CMF DB mapping file http://www.tdwg.org/schemas/abcd/2.06 was corrupt or could not be interpreted.</biocase:diagnostic> The requested schema (identified by the namespace http://www.tdwg.org/schemas/abcd/2.06) is not supported by that service. Create a mapping and map elements. <biocase:diagnostic severity="ERROR">The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts</biocase:diagnostic> Obvious - happens if you try to use a concept that is not mapped in a scan request or a filter. <biocase:diagnostic severity="ERROR">A SQL statement produced an error: SELECT unit.id, metadata.supplier_organisation, metadata.supplier_email, …, unit.febnr, unit.nameautor, unit.url_gbild, unit.datum1, unit.sammler, unit.fundort, unit.hoehe, metadata.id, unit.id FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) ORDER BY unit.id That could have a number of reasons: One of the table/column names that were set up in the configuration does not exist (because it was renamed or removed) or the credentials used by the BPS do not have sufficient privileges. Simply copy the SQL statement and execute it manually on the database with a regular database client that will show you the detailed error message returned by the DBMS. <biocase:diagnostic severity="WARNING">The mandatory child element SourceInstitutionID could not be created [0 existing, but 1 required]. The element /DataSets/DataSet/Units/Unit was also dropped.</biocase:diagnostic><biocase:diagnostic severity="WARNING">The mandatory child element Units could not be created [0 existing, but 1 required]. The element /DataSets/DataSet was also dropped.</biocase:diagnostic> One problem caused the next one in this sequence of warnings: The mandatory ABCD element was either not mapped or empty; therefore the respective record (“unit”) was dropped. Because this happened for all units in the datasets, all units were dropped, resulting in an empty Units sub tree. The second warning tells you what happened then: Because the Units element is mandatory for the Dataset element, the whole dataset was dropped. As a result, the response ABCD document was empty. == Debugging a BioCASe Installation == In case the debug output of the web service doesn’t help you to find the problem, or if the BPS behaves not as intended without displaying any error messages, you should have a look at the debug logs written by the BPS. To turn on the Debug logs, go to the System Administration page and set Debug logs to True: Bild Redo the step that didn’t work and open the log folder of your BioCASe installation. There are several global log files (pywrapper_request.log, pywrapper_error.log, webapp_debug.log, webapp_error.log) and a debug files per data source (e.g. debug_flora.log). Open these files and see if they can give you a clue to the problem. If you still can’t get it running, contact the __BioCASe Team__. ce6403607b2a4260e69640cfd212d6a44e224485 104 102 2011-06-30T15:19:02Z JoergHoletschek 3 /* Changing the Debug Level for a Data Source */ wikitext text/x-wiki This tutorial explains how to debug a BioCASe web service and a BioCASe installation in general. If you follow the instructions and don’t manage to get your service running, please contact the __BioCASe Team__. == Debugging a BioCASe Web Service == Debugging a BioCASe web service can be done with the manual Query Form. It allows you to load a BioCASe request (Search/Scan) from a template, edit the parameters of the request (e.g. the filter) and will show you the response document, which holds the requested ABCD records and useful debug information. === Opening the Query Form === There are several ways to open the Query Form. On the entry page of a datasource, <small>TEST and DEBUG</small> will open the Query Form: [[File:debugDatasourceEntryPage.png]] In the data source configuration tool, the link to the according query form can be always found in the link list at the top of the page. Moreover, in the mapping editor, the button <small>Test mapping!</small> will open the Query Form in a separate tab: [[File:abcdMappingEditorEmptyTop.png]] === Basic Usage === On top of the Query Form you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ''flora'' being the name of the datasource. Make sure this is the correct access point of the web service you want to debug! [[File:debugQueryformFull.png]] The usage of the Query Form is very straight forward: * Click on one of the links below the text box to load a template, for example for an ABCD2 Scan or ABCD2 Search; * Edit the request parameters; * Click on <small>Submit</small> to get the response document. The section <small>BioCASe protocol filter operators</small> lists the operators that can be used for creating complex filters that combine several filter criteria. More about creating filters can be found in the BioCASe Protocol documentation at http://www.biocase.org/products/protocols/index.shtml. Clicking on the <small>Show</small> link in the <small>Capabilities</small> section will show the Capabilities response in a frame of the Query Form. This is useful, since it shows the namespaces for the schemas supported by the web service (which is required in the request text) and XPaths of mapped ABCD elements (required for creating filters on these elements). By using Copy & Paste you can spare the trouble of entering these monstrous strings of characters. === Changing the Debug Level for a Data Source === Debug info can be set to one of four levels for a BioCASe data source: * <strong>Error</strong> is the highest level will only output errors that prevent the web server from answering a request. * <strong>Warning</strong> will output errors that keep BioCASe from answering a request and warnings that don’t prevent the fulfilling of the request, but should be analysed carefully. For example, they might tell you that records have been dropped because mandatory ABCD elements have not been mapped or were empty for certain records. * <strong>Info</strong> is the default and will add some more information about how the Provider Software processes the request. * <strong>Debug</strong> is the most detailed level and will produce very verbose debug output. It adds very comprehensive information about how the ABCD documents are constructed and is suitable only for experienced users. For most cases, keeping the default ''Info'' level should be sufficient for testing and debugging a web service. For production use, you should set it to ''Error'' later. You can do this in the configuration of a datasource under Settings: == Analyzing the Response Document == Upon clicking <small>Submit</small>, the Query Form will send the request to the web service and wait for the response. It might take some time – up to minutes – until the response document shows up. Amongst others, this depends on • the complexity of the query (especially the filters), • the size of the dataset returned, • the debug level set (“debug” results in very exhaustive debug output), • the performance of the servers running BioCASe and the database, • the network bandwidth. (bild) The response document consists of three sections. A full response document can be found here. === Header === This section contains information about the BioCASe installation – operating system, DBMS used, Python/BioCASe version etc. It is intended for a consumer application and won’t be very helpful for debugging: <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> === Content === This section makes up the payload of the response document, depending on the type of request: Capabilities: Lists all schemas supported by the web service. For each schema, all mapped concepts (identified by their XPaths) are listed with data type and searchable flag. Scan: The document will simply list all distinct values for the concept specified in the request. When done for the scientific name, it would look similar to this: <biocase:content recordCount="1664" recordDropped="0" recordStart="0" totalSearchHits="1664"><biocase:scan><biocase:value>Achillea atrata L.</biocase:value><biocase:value>Achillea millefolium L.</biocase:value><biocase:value>Achillea nobilis L. ssp. nobilis</biocase:value><biocase:value>Achillea ptarmica L.</biocase:value><biocase:value>Acinos alpinus (L.) Moench</biocase:value><biocase:value>Acinos arvensis (Lam.) Dandy</biocase:value><biocase:value>Aconitum degenii ssp. paniculatum (Arcang.) Mucher</biocase:value><biocase:value>Aconitum lycoctonum L. ssp. lycoctonum</biocase:value><biocase:value>Aconitum napellus L. s.l.</biocase:value> Search: For this type of request, the content consists of the ABCD dataset(s). See the ABCD2 example. === Debug Output === This section is the most interesting part for debugging. If debug level is set to Info (which is recommended for debugging), it will document the processing of the request by the web service. The output is easy to understand, for example it will contain the SQL statements sent to the database. For a successful search request it could look similar to this: == Sample Errors and Warnings in the Debug Output == <biocase:diagnostic severity="ERROR"> No DB connection could be established. Please verify DB state and connection parameters in your provider_setup_file.xml. </biocase:diagnostic> Well… just what it says. The Provider Software could not connect to the database to be published. Check the database connection settings (server IP, database name, user, and password) and make sure you’ve installed the Python package for the DBMS you’re using (see the Libs test page in Utilities). <biocase:diagnostic severity="INFO">Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06</biocase:diagnostic><biocase:diagnostic severity="INFO">Load CMFile 'C:\Workspace\bps2\config\datasources\flora\cmf_ABCD_2.06.xml'</biocase:diagnostic><biocase:diagnostic severity="ERROR">The CMF DB mapping file http://www.tdwg.org/schemas/abcd/2.06 was corrupt or could not be interpreted.</biocase:diagnostic> The requested schema (identified by the namespace http://www.tdwg.org/schemas/abcd/2.06) is not supported by that service. Create a mapping and map elements. <biocase:diagnostic severity="ERROR">The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts</biocase:diagnostic> Obvious - happens if you try to use a concept that is not mapped in a scan request or a filter. <biocase:diagnostic severity="ERROR">A SQL statement produced an error: SELECT unit.id, metadata.supplier_organisation, metadata.supplier_email, …, unit.febnr, unit.nameautor, unit.url_gbild, unit.datum1, unit.sammler, unit.fundort, unit.hoehe, metadata.id, unit.id FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) ORDER BY unit.id That could have a number of reasons: One of the table/column names that were set up in the configuration does not exist (because it was renamed or removed) or the credentials used by the BPS do not have sufficient privileges. Simply copy the SQL statement and execute it manually on the database with a regular database client that will show you the detailed error message returned by the DBMS. <biocase:diagnostic severity="WARNING">The mandatory child element SourceInstitutionID could not be created [0 existing, but 1 required]. The element /DataSets/DataSet/Units/Unit was also dropped.</biocase:diagnostic><biocase:diagnostic severity="WARNING">The mandatory child element Units could not be created [0 existing, but 1 required]. The element /DataSets/DataSet was also dropped.</biocase:diagnostic> One problem caused the next one in this sequence of warnings: The mandatory ABCD element was either not mapped or empty; therefore the respective record (“unit”) was dropped. Because this happened for all units in the datasets, all units were dropped, resulting in an empty Units sub tree. The second warning tells you what happened then: Because the Units element is mandatory for the Dataset element, the whole dataset was dropped. As a result, the response ABCD document was empty. == Debugging a BioCASe Installation == In case the debug output of the web service doesn’t help you to find the problem, or if the BPS behaves not as intended without displaying any error messages, you should have a look at the debug logs written by the BPS. To turn on the Debug logs, go to the System Administration page and set Debug logs to True: Bild Redo the step that didn’t work and open the log folder of your BioCASe installation. There are several global log files (pywrapper_request.log, pywrapper_error.log, webapp_debug.log, webapp_error.log) and a debug files per data source (e.g. debug_flora.log). Open these files and see if they can give you a clue to the problem. If you still can’t get it running, contact the __BioCASe Team__. 9f0160bdfce6f67791f9555f5fecb3f022373f38 105 104 2011-06-30T15:19:15Z JoergHoletschek 3 /* Debugging a BioCASe Installation */ wikitext text/x-wiki This tutorial explains how to debug a BioCASe web service and a BioCASe installation in general. If you follow the instructions and don’t manage to get your service running, please contact the __BioCASe Team__. == Debugging a BioCASe Web Service == Debugging a BioCASe web service can be done with the manual Query Form. It allows you to load a BioCASe request (Search/Scan) from a template, edit the parameters of the request (e.g. the filter) and will show you the response document, which holds the requested ABCD records and useful debug information. === Opening the Query Form === There are several ways to open the Query Form. On the entry page of a datasource, <small>TEST and DEBUG</small> will open the Query Form: [[File:debugDatasourceEntryPage.png]] In the data source configuration tool, the link to the according query form can be always found in the link list at the top of the page. Moreover, in the mapping editor, the button <small>Test mapping!</small> will open the Query Form in a separate tab: [[File:abcdMappingEditorEmptyTop.png]] === Basic Usage === On top of the Query Form you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ''flora'' being the name of the datasource. Make sure this is the correct access point of the web service you want to debug! [[File:debugQueryformFull.png]] The usage of the Query Form is very straight forward: * Click on one of the links below the text box to load a template, for example for an ABCD2 Scan or ABCD2 Search; * Edit the request parameters; * Click on <small>Submit</small> to get the response document. The section <small>BioCASe protocol filter operators</small> lists the operators that can be used for creating complex filters that combine several filter criteria. More about creating filters can be found in the BioCASe Protocol documentation at http://www.biocase.org/products/protocols/index.shtml. Clicking on the <small>Show</small> link in the <small>Capabilities</small> section will show the Capabilities response in a frame of the Query Form. This is useful, since it shows the namespaces for the schemas supported by the web service (which is required in the request text) and XPaths of mapped ABCD elements (required for creating filters on these elements). By using Copy & Paste you can spare the trouble of entering these monstrous strings of characters. === Changing the Debug Level for a Data Source === Debug info can be set to one of four levels for a BioCASe data source: * <strong>Error</strong> is the highest level will only output errors that prevent the web server from answering a request. * <strong>Warning</strong> will output errors that keep BioCASe from answering a request and warnings that don’t prevent the fulfilling of the request, but should be analysed carefully. For example, they might tell you that records have been dropped because mandatory ABCD elements have not been mapped or were empty for certain records. * <strong>Info</strong> is the default and will add some more information about how the Provider Software processes the request. * <strong>Debug</strong> is the most detailed level and will produce very verbose debug output. It adds very comprehensive information about how the ABCD documents are constructed and is suitable only for experienced users. For most cases, keeping the default ''Info'' level should be sufficient for testing and debugging a web service. For production use, you should set it to ''Error'' later. You can do this in the configuration of a datasource under Settings: == Analyzing the Response Document == Upon clicking <small>Submit</small>, the Query Form will send the request to the web service and wait for the response. It might take some time – up to minutes – until the response document shows up. Amongst others, this depends on • the complexity of the query (especially the filters), • the size of the dataset returned, • the debug level set (“debug” results in very exhaustive debug output), • the performance of the servers running BioCASe and the database, • the network bandwidth. (bild) The response document consists of three sections. A full response document can be found here. === Header === This section contains information about the BioCASe installation – operating system, DBMS used, Python/BioCASe version etc. It is intended for a consumer application and won’t be very helpful for debugging: <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> === Content === This section makes up the payload of the response document, depending on the type of request: Capabilities: Lists all schemas supported by the web service. For each schema, all mapped concepts (identified by their XPaths) are listed with data type and searchable flag. Scan: The document will simply list all distinct values for the concept specified in the request. When done for the scientific name, it would look similar to this: <biocase:content recordCount="1664" recordDropped="0" recordStart="0" totalSearchHits="1664"><biocase:scan><biocase:value>Achillea atrata L.</biocase:value><biocase:value>Achillea millefolium L.</biocase:value><biocase:value>Achillea nobilis L. ssp. nobilis</biocase:value><biocase:value>Achillea ptarmica L.</biocase:value><biocase:value>Acinos alpinus (L.) Moench</biocase:value><biocase:value>Acinos arvensis (Lam.) Dandy</biocase:value><biocase:value>Aconitum degenii ssp. paniculatum (Arcang.) Mucher</biocase:value><biocase:value>Aconitum lycoctonum L. ssp. lycoctonum</biocase:value><biocase:value>Aconitum napellus L. s.l.</biocase:value> Search: For this type of request, the content consists of the ABCD dataset(s). See the ABCD2 example. === Debug Output === This section is the most interesting part for debugging. If debug level is set to Info (which is recommended for debugging), it will document the processing of the request by the web service. The output is easy to understand, for example it will contain the SQL statements sent to the database. For a successful search request it could look similar to this: == Sample Errors and Warnings in the Debug Output == <biocase:diagnostic severity="ERROR"> No DB connection could be established. Please verify DB state and connection parameters in your provider_setup_file.xml. </biocase:diagnostic> Well… just what it says. The Provider Software could not connect to the database to be published. Check the database connection settings (server IP, database name, user, and password) and make sure you’ve installed the Python package for the DBMS you’re using (see the Libs test page in Utilities). <biocase:diagnostic severity="INFO">Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06</biocase:diagnostic><biocase:diagnostic severity="INFO">Load CMFile 'C:\Workspace\bps2\config\datasources\flora\cmf_ABCD_2.06.xml'</biocase:diagnostic><biocase:diagnostic severity="ERROR">The CMF DB mapping file http://www.tdwg.org/schemas/abcd/2.06 was corrupt or could not be interpreted.</biocase:diagnostic> The requested schema (identified by the namespace http://www.tdwg.org/schemas/abcd/2.06) is not supported by that service. Create a mapping and map elements. <biocase:diagnostic severity="ERROR">The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts</biocase:diagnostic> Obvious - happens if you try to use a concept that is not mapped in a scan request or a filter. <biocase:diagnostic severity="ERROR">A SQL statement produced an error: SELECT unit.id, metadata.supplier_organisation, metadata.supplier_email, …, unit.febnr, unit.nameautor, unit.url_gbild, unit.datum1, unit.sammler, unit.fundort, unit.hoehe, metadata.id, unit.id FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) ORDER BY unit.id That could have a number of reasons: One of the table/column names that were set up in the configuration does not exist (because it was renamed or removed) or the credentials used by the BPS do not have sufficient privileges. Simply copy the SQL statement and execute it manually on the database with a regular database client that will show you the detailed error message returned by the DBMS. <biocase:diagnostic severity="WARNING">The mandatory child element SourceInstitutionID could not be created [0 existing, but 1 required]. The element /DataSets/DataSet/Units/Unit was also dropped.</biocase:diagnostic><biocase:diagnostic severity="WARNING">The mandatory child element Units could not be created [0 existing, but 1 required]. The element /DataSets/DataSet was also dropped.</biocase:diagnostic> One problem caused the next one in this sequence of warnings: The mandatory ABCD element was either not mapped or empty; therefore the respective record (“unit”) was dropped. Because this happened for all units in the datasets, all units were dropped, resulting in an empty Units sub tree. The second warning tells you what happened then: Because the Units element is mandatory for the Dataset element, the whole dataset was dropped. As a result, the response ABCD document was empty. == Debugging a BioCASe Installation == In case the debug output of the web service doesn’t help you to find the problem, or if the BPS behaves not as intended without displaying any error messages, you should have a look at the debug logs written by the BPS. To turn on the Debug logs, go to the System Administration page and set Debug logs to True: [[File:instConfigTool.png|500px]] Redo the step that didn’t work and open the log folder of your BioCASe installation. There are several global log files (pywrapper_request.log, pywrapper_error.log, webapp_debug.log, webapp_error.log) and a debug files per data source (e.g. debug_flora.log). Open these files and see if they can give you a clue to the problem. If you still can’t get it running, contact the __BioCASe Team__. c1b42d7681d5032807f0a60cbc0a4ecf660c7a0a 106 105 2011-06-30T15:20:18Z JoergHoletschek 3 /* Changing the Debug Level for a Data Source */ wikitext text/x-wiki This tutorial explains how to debug a BioCASe web service and a BioCASe installation in general. If you follow the instructions and don’t manage to get your service running, please contact the __BioCASe Team__. == Debugging a BioCASe Web Service == Debugging a BioCASe web service can be done with the manual Query Form. It allows you to load a BioCASe request (Search/Scan) from a template, edit the parameters of the request (e.g. the filter) and will show you the response document, which holds the requested ABCD records and useful debug information. === Opening the Query Form === There are several ways to open the Query Form. On the entry page of a datasource, <small>TEST and DEBUG</small> will open the Query Form: [[File:debugDatasourceEntryPage.png]] In the data source configuration tool, the link to the according query form can be always found in the link list at the top of the page. Moreover, in the mapping editor, the button <small>Test mapping!</small> will open the Query Form in a separate tab: [[File:abcdMappingEditorEmptyTop.png]] === Basic Usage === On top of the Query Form you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ''flora'' being the name of the datasource. Make sure this is the correct access point of the web service you want to debug! [[File:debugQueryformFull.png]] The usage of the Query Form is very straight forward: * Click on one of the links below the text box to load a template, for example for an ABCD2 Scan or ABCD2 Search; * Edit the request parameters; * Click on <small>Submit</small> to get the response document. The section <small>BioCASe protocol filter operators</small> lists the operators that can be used for creating complex filters that combine several filter criteria. More about creating filters can be found in the BioCASe Protocol documentation at http://www.biocase.org/products/protocols/index.shtml. Clicking on the <small>Show</small> link in the <small>Capabilities</small> section will show the Capabilities response in a frame of the Query Form. This is useful, since it shows the namespaces for the schemas supported by the web service (which is required in the request text) and XPaths of mapped ABCD elements (required for creating filters on these elements). By using Copy & Paste you can spare the trouble of entering these monstrous strings of characters. === Changing the Debug Level for a Data Source === Debug info can be set to one of four levels for a BioCASe data source: * <strong>Error</strong> is the highest level will only output errors that prevent the web server from answering a request. * <strong>Warning</strong> will output errors that keep BioCASe from answering a request and warnings that don’t prevent the fulfilling of the request, but should be analysed carefully. For example, they might tell you that records have been dropped because mandatory ABCD elements have not been mapped or were empty for certain records. * <strong>Info</strong> is the default and will add some more information about how the Provider Software processes the request. * <strong>Debug</strong> is the most detailed level and will produce very verbose debug output. It adds very comprehensive information about how the ABCD documents are constructed and is suitable only for experienced users. For most cases, keeping the default ''Info'' level should be sufficient for testing and debugging a web service. For production use, you should set it to ''Error'' later. You can do this in the configuration of a datasource under Settings: [[File:debugDatasourceSettings.png]] == Analyzing the Response Document == Upon clicking <small>Submit</small>, the Query Form will send the request to the web service and wait for the response. It might take some time – up to minutes – until the response document shows up. Amongst others, this depends on • the complexity of the query (especially the filters), • the size of the dataset returned, • the debug level set (“debug” results in very exhaustive debug output), • the performance of the servers running BioCASe and the database, • the network bandwidth. (bild) The response document consists of three sections. A full response document can be found here. === Header === This section contains information about the BioCASe installation – operating system, DBMS used, Python/BioCASe version etc. It is intended for a consumer application and won’t be very helpful for debugging: <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> === Content === This section makes up the payload of the response document, depending on the type of request: Capabilities: Lists all schemas supported by the web service. For each schema, all mapped concepts (identified by their XPaths) are listed with data type and searchable flag. Scan: The document will simply list all distinct values for the concept specified in the request. When done for the scientific name, it would look similar to this: <biocase:content recordCount="1664" recordDropped="0" recordStart="0" totalSearchHits="1664"><biocase:scan><biocase:value>Achillea atrata L.</biocase:value><biocase:value>Achillea millefolium L.</biocase:value><biocase:value>Achillea nobilis L. ssp. nobilis</biocase:value><biocase:value>Achillea ptarmica L.</biocase:value><biocase:value>Acinos alpinus (L.) Moench</biocase:value><biocase:value>Acinos arvensis (Lam.) Dandy</biocase:value><biocase:value>Aconitum degenii ssp. paniculatum (Arcang.) Mucher</biocase:value><biocase:value>Aconitum lycoctonum L. ssp. lycoctonum</biocase:value><biocase:value>Aconitum napellus L. s.l.</biocase:value> Search: For this type of request, the content consists of the ABCD dataset(s). See the ABCD2 example. === Debug Output === This section is the most interesting part for debugging. If debug level is set to Info (which is recommended for debugging), it will document the processing of the request by the web service. The output is easy to understand, for example it will contain the SQL statements sent to the database. For a successful search request it could look similar to this: == Sample Errors and Warnings in the Debug Output == <biocase:diagnostic severity="ERROR"> No DB connection could be established. Please verify DB state and connection parameters in your provider_setup_file.xml. </biocase:diagnostic> Well… just what it says. The Provider Software could not connect to the database to be published. Check the database connection settings (server IP, database name, user, and password) and make sure you’ve installed the Python package for the DBMS you’re using (see the Libs test page in Utilities). <biocase:diagnostic severity="INFO">Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06</biocase:diagnostic><biocase:diagnostic severity="INFO">Load CMFile 'C:\Workspace\bps2\config\datasources\flora\cmf_ABCD_2.06.xml'</biocase:diagnostic><biocase:diagnostic severity="ERROR">The CMF DB mapping file http://www.tdwg.org/schemas/abcd/2.06 was corrupt or could not be interpreted.</biocase:diagnostic> The requested schema (identified by the namespace http://www.tdwg.org/schemas/abcd/2.06) is not supported by that service. Create a mapping and map elements. <biocase:diagnostic severity="ERROR">The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts</biocase:diagnostic> Obvious - happens if you try to use a concept that is not mapped in a scan request or a filter. <biocase:diagnostic severity="ERROR">A SQL statement produced an error: SELECT unit.id, metadata.supplier_organisation, metadata.supplier_email, …, unit.febnr, unit.nameautor, unit.url_gbild, unit.datum1, unit.sammler, unit.fundort, unit.hoehe, metadata.id, unit.id FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) ORDER BY unit.id That could have a number of reasons: One of the table/column names that were set up in the configuration does not exist (because it was renamed or removed) or the credentials used by the BPS do not have sufficient privileges. Simply copy the SQL statement and execute it manually on the database with a regular database client that will show you the detailed error message returned by the DBMS. <biocase:diagnostic severity="WARNING">The mandatory child element SourceInstitutionID could not be created [0 existing, but 1 required]. The element /DataSets/DataSet/Units/Unit was also dropped.</biocase:diagnostic><biocase:diagnostic severity="WARNING">The mandatory child element Units could not be created [0 existing, but 1 required]. The element /DataSets/DataSet was also dropped.</biocase:diagnostic> One problem caused the next one in this sequence of warnings: The mandatory ABCD element was either not mapped or empty; therefore the respective record (“unit”) was dropped. Because this happened for all units in the datasets, all units were dropped, resulting in an empty Units sub tree. The second warning tells you what happened then: Because the Units element is mandatory for the Dataset element, the whole dataset was dropped. As a result, the response ABCD document was empty. == Debugging a BioCASe Installation == In case the debug output of the web service doesn’t help you to find the problem, or if the BPS behaves not as intended without displaying any error messages, you should have a look at the debug logs written by the BPS. To turn on the Debug logs, go to the System Administration page and set Debug logs to True: [[File:instConfigTool.png|500px]] Redo the step that didn’t work and open the log folder of your BioCASe installation. There are several global log files (pywrapper_request.log, pywrapper_error.log, webapp_debug.log, webapp_error.log) and a debug files per data source (e.g. debug_flora.log). Open these files and see if they can give you a clue to the problem. If you still can’t get it running, contact the __BioCASe Team__. 975f631174c4d9ae00b473202495fd12e25e6813 108 106 2011-06-30T15:23:33Z JoergHoletschek 3 /* Analyzing the Response Document */ wikitext text/x-wiki This tutorial explains how to debug a BioCASe web service and a BioCASe installation in general. If you follow the instructions and don’t manage to get your service running, please contact the __BioCASe Team__. == Debugging a BioCASe Web Service == Debugging a BioCASe web service can be done with the manual Query Form. It allows you to load a BioCASe request (Search/Scan) from a template, edit the parameters of the request (e.g. the filter) and will show you the response document, which holds the requested ABCD records and useful debug information. === Opening the Query Form === There are several ways to open the Query Form. On the entry page of a datasource, <small>TEST and DEBUG</small> will open the Query Form: [[File:debugDatasourceEntryPage.png]] In the data source configuration tool, the link to the according query form can be always found in the link list at the top of the page. Moreover, in the mapping editor, the button <small>Test mapping!</small> will open the Query Form in a separate tab: [[File:abcdMappingEditorEmptyTop.png]] === Basic Usage === On top of the Query Form you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ''flora'' being the name of the datasource. Make sure this is the correct access point of the web service you want to debug! [[File:debugQueryformFull.png]] The usage of the Query Form is very straight forward: * Click on one of the links below the text box to load a template, for example for an ABCD2 Scan or ABCD2 Search; * Edit the request parameters; * Click on <small>Submit</small> to get the response document. The section <small>BioCASe protocol filter operators</small> lists the operators that can be used for creating complex filters that combine several filter criteria. More about creating filters can be found in the BioCASe Protocol documentation at http://www.biocase.org/products/protocols/index.shtml. Clicking on the <small>Show</small> link in the <small>Capabilities</small> section will show the Capabilities response in a frame of the Query Form. This is useful, since it shows the namespaces for the schemas supported by the web service (which is required in the request text) and XPaths of mapped ABCD elements (required for creating filters on these elements). By using Copy & Paste you can spare the trouble of entering these monstrous strings of characters. === Changing the Debug Level for a Data Source === Debug info can be set to one of four levels for a BioCASe data source: * <strong>Error</strong> is the highest level will only output errors that prevent the web server from answering a request. * <strong>Warning</strong> will output errors that keep BioCASe from answering a request and warnings that don’t prevent the fulfilling of the request, but should be analysed carefully. For example, they might tell you that records have been dropped because mandatory ABCD elements have not been mapped or were empty for certain records. * <strong>Info</strong> is the default and will add some more information about how the Provider Software processes the request. * <strong>Debug</strong> is the most detailed level and will produce very verbose debug output. It adds very comprehensive information about how the ABCD documents are constructed and is suitable only for experienced users. For most cases, keeping the default ''Info'' level should be sufficient for testing and debugging a web service. For production use, you should set it to ''Error'' later. You can do this in the configuration of a datasource under Settings: [[File:debugDatasourceSettings.png]] == Analyzing the Response Document == Upon clicking <small>Submit</small>, the Query Form will send the request to the web service and wait for the response. It might take some time – up to minutes – until the response document shows up. Amongst others, this depends on * the complexity of the query (especially the filters), * the size of the dataset returned, * the debug level set (“debug” results in very exhaustive debug output), * the performance of the servers running BioCASe and the database, * the network bandwidth. The response document consists of three sections (a full response document can be found [[CommonABCD2Concepts|here]]): [[File:debugResponseDocumentOverview.png]] === Header === This section contains information about the BioCASe installation – operating system, DBMS used, Python/BioCASe version etc. It is intended for a consumer application and won’t be very helpful for debugging: <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> === Content === This section makes up the payload of the response document, depending on the type of request: Capabilities: Lists all schemas supported by the web service. For each schema, all mapped concepts (identified by their XPaths) are listed with data type and searchable flag. Scan: The document will simply list all distinct values for the concept specified in the request. When done for the scientific name, it would look similar to this: <biocase:content recordCount="1664" recordDropped="0" recordStart="0" totalSearchHits="1664"><biocase:scan><biocase:value>Achillea atrata L.</biocase:value><biocase:value>Achillea millefolium L.</biocase:value><biocase:value>Achillea nobilis L. ssp. nobilis</biocase:value><biocase:value>Achillea ptarmica L.</biocase:value><biocase:value>Acinos alpinus (L.) Moench</biocase:value><biocase:value>Acinos arvensis (Lam.) Dandy</biocase:value><biocase:value>Aconitum degenii ssp. paniculatum (Arcang.) Mucher</biocase:value><biocase:value>Aconitum lycoctonum L. ssp. lycoctonum</biocase:value><biocase:value>Aconitum napellus L. s.l.</biocase:value> Search: For this type of request, the content consists of the ABCD dataset(s). See the ABCD2 example. === Debug Output === This section is the most interesting part for debugging. If debug level is set to Info (which is recommended for debugging), it will document the processing of the request by the web service. The output is easy to understand, for example it will contain the SQL statements sent to the database. For a successful search request it could look similar to this: == Sample Errors and Warnings in the Debug Output == <biocase:diagnostic severity="ERROR"> No DB connection could be established. Please verify DB state and connection parameters in your provider_setup_file.xml. </biocase:diagnostic> Well… just what it says. The Provider Software could not connect to the database to be published. Check the database connection settings (server IP, database name, user, and password) and make sure you’ve installed the Python package for the DBMS you’re using (see the Libs test page in Utilities). <biocase:diagnostic severity="INFO">Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06</biocase:diagnostic><biocase:diagnostic severity="INFO">Load CMFile 'C:\Workspace\bps2\config\datasources\flora\cmf_ABCD_2.06.xml'</biocase:diagnostic><biocase:diagnostic severity="ERROR">The CMF DB mapping file http://www.tdwg.org/schemas/abcd/2.06 was corrupt or could not be interpreted.</biocase:diagnostic> The requested schema (identified by the namespace http://www.tdwg.org/schemas/abcd/2.06) is not supported by that service. Create a mapping and map elements. <biocase:diagnostic severity="ERROR">The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts</biocase:diagnostic> Obvious - happens if you try to use a concept that is not mapped in a scan request or a filter. <biocase:diagnostic severity="ERROR">A SQL statement produced an error: SELECT unit.id, metadata.supplier_organisation, metadata.supplier_email, …, unit.febnr, unit.nameautor, unit.url_gbild, unit.datum1, unit.sammler, unit.fundort, unit.hoehe, metadata.id, unit.id FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) ORDER BY unit.id That could have a number of reasons: One of the table/column names that were set up in the configuration does not exist (because it was renamed or removed) or the credentials used by the BPS do not have sufficient privileges. Simply copy the SQL statement and execute it manually on the database with a regular database client that will show you the detailed error message returned by the DBMS. <biocase:diagnostic severity="WARNING">The mandatory child element SourceInstitutionID could not be created [0 existing, but 1 required]. The element /DataSets/DataSet/Units/Unit was also dropped.</biocase:diagnostic><biocase:diagnostic severity="WARNING">The mandatory child element Units could not be created [0 existing, but 1 required]. The element /DataSets/DataSet was also dropped.</biocase:diagnostic> One problem caused the next one in this sequence of warnings: The mandatory ABCD element was either not mapped or empty; therefore the respective record (“unit”) was dropped. Because this happened for all units in the datasets, all units were dropped, resulting in an empty Units sub tree. The second warning tells you what happened then: Because the Units element is mandatory for the Dataset element, the whole dataset was dropped. As a result, the response ABCD document was empty. == Debugging a BioCASe Installation == In case the debug output of the web service doesn’t help you to find the problem, or if the BPS behaves not as intended without displaying any error messages, you should have a look at the debug logs written by the BPS. To turn on the Debug logs, go to the System Administration page and set Debug logs to True: [[File:instConfigTool.png|500px]] Redo the step that didn’t work and open the log folder of your BioCASe installation. There are several global log files (pywrapper_request.log, pywrapper_error.log, webapp_debug.log, webapp_error.log) and a debug files per data source (e.g. debug_flora.log). Open these files and see if they can give you a clue to the problem. If you still can’t get it running, contact the __BioCASe Team__. 3e92e99cbd6bd366a821fd47f806bef4303ae489 110 108 2011-06-30T15:41:07Z JoergHoletschek 3 /* Analyzing the Response Document */ wikitext text/x-wiki This tutorial explains how to debug a BioCASe web service and a BioCASe installation in general. If you follow the instructions and don’t manage to get your service running, please contact the __BioCASe Team__. == Debugging a BioCASe Web Service == Debugging a BioCASe web service can be done with the manual Query Form. It allows you to load a BioCASe request (Search/Scan) from a template, edit the parameters of the request (e.g. the filter) and will show you the response document, which holds the requested ABCD records and useful debug information. === Opening the Query Form === There are several ways to open the Query Form. On the entry page of a datasource, <small>TEST and DEBUG</small> will open the Query Form: [[File:debugDatasourceEntryPage.png]] In the data source configuration tool, the link to the according query form can be always found in the link list at the top of the page. Moreover, in the mapping editor, the button <small>Test mapping!</small> will open the Query Form in a separate tab: [[File:abcdMappingEditorEmptyTop.png]] === Basic Usage === On top of the Query Form you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ''flora'' being the name of the datasource. Make sure this is the correct access point of the web service you want to debug! [[File:debugQueryformFull.png]] The usage of the Query Form is very straight forward: * Click on one of the links below the text box to load a template, for example for an ABCD2 Scan or ABCD2 Search; * Edit the request parameters; * Click on <small>Submit</small> to get the response document. The section <small>BioCASe protocol filter operators</small> lists the operators that can be used for creating complex filters that combine several filter criteria. More about creating filters can be found in the BioCASe Protocol documentation at http://www.biocase.org/products/protocols/index.shtml. Clicking on the <small>Show</small> link in the <small>Capabilities</small> section will show the Capabilities response in a frame of the Query Form. This is useful, since it shows the namespaces for the schemas supported by the web service (which is required in the request text) and XPaths of mapped ABCD elements (required for creating filters on these elements). By using Copy & Paste you can spare the trouble of entering these monstrous strings of characters. === Changing the Debug Level for a Data Source === Debug info can be set to one of four levels for a BioCASe data source: * <strong>Error</strong> is the highest level will only output errors that prevent the web server from answering a request. * <strong>Warning</strong> will output errors that keep BioCASe from answering a request and warnings that don’t prevent the fulfilling of the request, but should be analysed carefully. For example, they might tell you that records have been dropped because mandatory ABCD elements have not been mapped or were empty for certain records. * <strong>Info</strong> is the default and will add some more information about how the Provider Software processes the request. * <strong>Debug</strong> is the most detailed level and will produce very verbose debug output. It adds very comprehensive information about how the ABCD documents are constructed and is suitable only for experienced users. For most cases, keeping the default ''Info'' level should be sufficient for testing and debugging a web service. For production use, you should set it to ''Error'' later. You can do this in the configuration of a datasource under Settings: [[File:debugDatasourceSettings.png]] == Analyzing the Response Document == Upon clicking <small>Submit</small>, the Query Form will send the request to the web service and wait for the response. It might take some time – up to minutes – until the response document shows up. Amongst others, this depends on * the complexity of the query (especially the filters), * the size of the dataset returned, * the debug level set (“debug” results in very exhaustive debug output), * the performance of the servers running BioCASe and the database, * the network bandwidth. The response document consists of three sections (a full response document can be found [[CommonABCD2Concepts|here]]): [[File:debugResponseDocumentOverview.png]] === Header === This section contains information about the BioCASe installation – operating system, DBMS used, Python/BioCASe version etc. It is intended for a consumer application and won’t be very helpful for debugging: <pre> <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> </pre> === Content === This section makes up the payload of the response document, depending on the type of request: <strong>Capabilities:</strong> Lists all schemas supported by the web service. For each schema, all mapped concepts (identified by their XPaths) are listed with data type and searchable flag. <strong>Scan:</strong> The document will simply list all distinct values for the concept specified in the request. When done for the scientific name, it would look similar to this: <pre> <biocase:content recordCount="1664" recordDropped="0" recordStart="0" totalSearchHits="1664"> <biocase:scan> <biocase:value>Achillea atrata L.</biocase:value> <biocase:value>Achillea millefolium L.</biocase:value> <biocase:value>Achillea nobilis L. ssp. nobilis</biocase:value> <biocase:value>Achillea ptarmica L.</biocase:value> <biocase:value>Acinos alpinus (L.) Moench</biocase:value> <biocase:value>Acinos arvensis (Lam.) Dandy</biocase:value> <biocase:value>Aconitum degenii ssp. paniculatum (Arcang.) Mucher</biocase:value> <biocase:value>Aconitum lycoctonum L. ssp. lycoctonum</biocase:value> <biocase:value>Aconitum napellus L. s.l.</biocase:value> ... </biocase:scan> </biocase:content> </pre> <strong>Search:</strong> For this type of request, the content consists of the ABCD dataset(s). See the the [[CommonABCD2Concepts| ABCD2 example]]. === Debug Output === This section is the most interesting part for debugging. If debug level is set to ''Info'' (which is recommended for debugging), it will document the processing of the request by the web service. The output is easy to understand, for example it will contain the SQL statements sent to the database. For a successful search request it could look similar to this: <pre> <biocase:diagnostics> <biocase:diagnostic severity="INFO"> Datasource wrapper FloraExsiccataBavarica requested </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Reading PSF from C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\provider_setup_file.xml </biocase:diagnostic> <biocase:diagnostic severity="DEBUG"> PSF: PSF=C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\provider_setup_file.xml, recLimit=100, loglevel=20, user=webuser, database=rbg, dbIP=192.168.2.10, dbms=mysql, encoding=latin_1, schemas={u'http://www.tdwg.org/schemas/abcd/2.06': <biocase.wrapper.psf_handler.SupportedSchema instance at 0x011F5B98>}, tablegraph=GRAPH: graph: unit-metadata, +++ ALIAS2TABLE: {u'unit': u'feb3', u'metadata': u'metadata'} </biocase:diagnostic> <biocase:diagnostic severity="INFO"> BioCASe protocol used. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> HTTP parameter 'query' used for building the request. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT DISTINCT unit.id FROM feb3 AS unit WHERE (unit.nameautor LIKE 'Allium %' ) LIMIT 11' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT unit.id, metadata.supplier_organisation, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (40, 41, 42, 43, 44, 45, 46, 47)) ORDER BY unit.id' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> time to execute request is 0.46599984169 </biocase:diagnostic> </biocase:diagnostics> </pre> == Sample Errors and Warnings in the Debug Output == <biocase:diagnostic severity="ERROR"> No DB connection could be established. Please verify DB state and connection parameters in your provider_setup_file.xml. </biocase:diagnostic> Well… just what it says. The Provider Software could not connect to the database to be published. Check the database connection settings (server IP, database name, user, and password) and make sure you’ve installed the Python package for the DBMS you’re using (see the Libs test page in Utilities). <biocase:diagnostic severity="INFO">Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06</biocase:diagnostic><biocase:diagnostic severity="INFO">Load CMFile 'C:\Workspace\bps2\config\datasources\flora\cmf_ABCD_2.06.xml'</biocase:diagnostic><biocase:diagnostic severity="ERROR">The CMF DB mapping file http://www.tdwg.org/schemas/abcd/2.06 was corrupt or could not be interpreted.</biocase:diagnostic> The requested schema (identified by the namespace http://www.tdwg.org/schemas/abcd/2.06) is not supported by that service. Create a mapping and map elements. <biocase:diagnostic severity="ERROR">The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts</biocase:diagnostic> Obvious - happens if you try to use a concept that is not mapped in a scan request or a filter. <biocase:diagnostic severity="ERROR">A SQL statement produced an error: SELECT unit.id, metadata.supplier_organisation, metadata.supplier_email, …, unit.febnr, unit.nameautor, unit.url_gbild, unit.datum1, unit.sammler, unit.fundort, unit.hoehe, metadata.id, unit.id FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) ORDER BY unit.id That could have a number of reasons: One of the table/column names that were set up in the configuration does not exist (because it was renamed or removed) or the credentials used by the BPS do not have sufficient privileges. Simply copy the SQL statement and execute it manually on the database with a regular database client that will show you the detailed error message returned by the DBMS. <biocase:diagnostic severity="WARNING">The mandatory child element SourceInstitutionID could not be created [0 existing, but 1 required]. The element /DataSets/DataSet/Units/Unit was also dropped.</biocase:diagnostic><biocase:diagnostic severity="WARNING">The mandatory child element Units could not be created [0 existing, but 1 required]. The element /DataSets/DataSet was also dropped.</biocase:diagnostic> One problem caused the next one in this sequence of warnings: The mandatory ABCD element was either not mapped or empty; therefore the respective record (“unit”) was dropped. Because this happened for all units in the datasets, all units were dropped, resulting in an empty Units sub tree. The second warning tells you what happened then: Because the Units element is mandatory for the Dataset element, the whole dataset was dropped. As a result, the response ABCD document was empty. == Debugging a BioCASe Installation == In case the debug output of the web service doesn’t help you to find the problem, or if the BPS behaves not as intended without displaying any error messages, you should have a look at the debug logs written by the BPS. To turn on the Debug logs, go to the System Administration page and set Debug logs to True: [[File:instConfigTool.png|500px]] Redo the step that didn’t work and open the log folder of your BioCASe installation. There are several global log files (pywrapper_request.log, pywrapper_error.log, webapp_debug.log, webapp_error.log) and a debug files per data source (e.g. debug_flora.log). Open these files and see if they can give you a clue to the problem. If you still can’t get it running, contact the __BioCASe Team__. 3f48a4478e5be1afb26707d7e793e07a7d0ea393 111 110 2011-06-30T15:42:02Z JoergHoletschek 3 /* Debug Output */ wikitext text/x-wiki This tutorial explains how to debug a BioCASe web service and a BioCASe installation in general. If you follow the instructions and don’t manage to get your service running, please contact the __BioCASe Team__. == Debugging a BioCASe Web Service == Debugging a BioCASe web service can be done with the manual Query Form. It allows you to load a BioCASe request (Search/Scan) from a template, edit the parameters of the request (e.g. the filter) and will show you the response document, which holds the requested ABCD records and useful debug information. === Opening the Query Form === There are several ways to open the Query Form. On the entry page of a datasource, <small>TEST and DEBUG</small> will open the Query Form: [[File:debugDatasourceEntryPage.png]] In the data source configuration tool, the link to the according query form can be always found in the link list at the top of the page. Moreover, in the mapping editor, the button <small>Test mapping!</small> will open the Query Form in a separate tab: [[File:abcdMappingEditorEmptyTop.png]] === Basic Usage === On top of the Query Form you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ''flora'' being the name of the datasource. Make sure this is the correct access point of the web service you want to debug! [[File:debugQueryformFull.png]] The usage of the Query Form is very straight forward: * Click on one of the links below the text box to load a template, for example for an ABCD2 Scan or ABCD2 Search; * Edit the request parameters; * Click on <small>Submit</small> to get the response document. The section <small>BioCASe protocol filter operators</small> lists the operators that can be used for creating complex filters that combine several filter criteria. More about creating filters can be found in the BioCASe Protocol documentation at http://www.biocase.org/products/protocols/index.shtml. Clicking on the <small>Show</small> link in the <small>Capabilities</small> section will show the Capabilities response in a frame of the Query Form. This is useful, since it shows the namespaces for the schemas supported by the web service (which is required in the request text) and XPaths of mapped ABCD elements (required for creating filters on these elements). By using Copy & Paste you can spare the trouble of entering these monstrous strings of characters. === Changing the Debug Level for a Data Source === Debug info can be set to one of four levels for a BioCASe data source: * <strong>Error</strong> is the highest level will only output errors that prevent the web server from answering a request. * <strong>Warning</strong> will output errors that keep BioCASe from answering a request and warnings that don’t prevent the fulfilling of the request, but should be analysed carefully. For example, they might tell you that records have been dropped because mandatory ABCD elements have not been mapped or were empty for certain records. * <strong>Info</strong> is the default and will add some more information about how the Provider Software processes the request. * <strong>Debug</strong> is the most detailed level and will produce very verbose debug output. It adds very comprehensive information about how the ABCD documents are constructed and is suitable only for experienced users. For most cases, keeping the default ''Info'' level should be sufficient for testing and debugging a web service. For production use, you should set it to ''Error'' later. You can do this in the configuration of a datasource under Settings: [[File:debugDatasourceSettings.png]] == Analyzing the Response Document == Upon clicking <small>Submit</small>, the Query Form will send the request to the web service and wait for the response. It might take some time – up to minutes – until the response document shows up. Amongst others, this depends on * the complexity of the query (especially the filters), * the size of the dataset returned, * the debug level set (“debug” results in very exhaustive debug output), * the performance of the servers running BioCASe and the database, * the network bandwidth. The response document consists of three sections (a full response document can be found [[CommonABCD2Concepts|here]]): [[File:debugResponseDocumentOverview.png]] === Header === This section contains information about the BioCASe installation – operating system, DBMS used, Python/BioCASe version etc. It is intended for a consumer application and won’t be very helpful for debugging: <pre> <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> </pre> === Content === This section makes up the payload of the response document, depending on the type of request: <strong>Capabilities:</strong> Lists all schemas supported by the web service. For each schema, all mapped concepts (identified by their XPaths) are listed with data type and searchable flag. <strong>Scan:</strong> The document will simply list all distinct values for the concept specified in the request. When done for the scientific name, it would look similar to this: <pre> <biocase:content recordCount="1664" recordDropped="0" recordStart="0" totalSearchHits="1664"> <biocase:scan> <biocase:value>Achillea atrata L.</biocase:value> <biocase:value>Achillea millefolium L.</biocase:value> <biocase:value>Achillea nobilis L. ssp. nobilis</biocase:value> <biocase:value>Achillea ptarmica L.</biocase:value> <biocase:value>Acinos alpinus (L.) Moench</biocase:value> <biocase:value>Acinos arvensis (Lam.) Dandy</biocase:value> <biocase:value>Aconitum degenii ssp. paniculatum (Arcang.) Mucher</biocase:value> <biocase:value>Aconitum lycoctonum L. ssp. lycoctonum</biocase:value> <biocase:value>Aconitum napellus L. s.l.</biocase:value> ... </biocase:scan> </biocase:content> </pre> <strong>Search:</strong> For this type of request, the content consists of the ABCD dataset(s). See the the [[CommonABCD2Concepts| ABCD2 example]]. === Debug Output === This section is the most interesting part for debugging. If debug level is set to ''Info'' (which is recommended for debugging), it will document the processing of the request by the web service. The output is easy to understand, for example it will contain the SQL statements sent to the database. For a successful search request it could look similar to this: <pre> <biocase:diagnostics> <biocase:diagnostic severity="INFO"> Datasource wrapper FloraExsiccataBavarica requested </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Reading PSF from C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica \provider_setup_file.xml </biocase:diagnostic> <biocase:diagnostic severity="DEBUG"> PSF: PSF=C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\provider_setup_file.xml, recLimit=100, loglevel=20, user=webuser, database=rbg, dbIP=192.168.2.10, dbms=mysql, encoding=latin_1, schemas={u'http://www.tdwg.org/schemas/abcd/2.06': <biocase.wrapper.psf_handler.SupportedSchema instance at 0x011F5B98>}, tablegraph=GRAPH: graph: unit-metadata, +++ ALIAS2TABLE: {u'unit': u'feb3', u'metadata': u'metadata'} </biocase:diagnostic> <biocase:diagnostic severity="INFO"> BioCASe protocol used. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> HTTP parameter 'query' used for building the request. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT DISTINCT unit.id FROM feb3 AS unit WHERE (unit.nameautor LIKE 'Allium %' ) LIMIT 11' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT unit.id, metadata.supplier_organisation, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (40, 41, 42, 43, 44, 45, 46, 47)) ORDER BY unit.id' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> time to execute request is 0.46599984169 </biocase:diagnostic> </biocase:diagnostics> </pre> == Sample Errors and Warnings in the Debug Output == <biocase:diagnostic severity="ERROR"> No DB connection could be established. Please verify DB state and connection parameters in your provider_setup_file.xml. </biocase:diagnostic> Well… just what it says. The Provider Software could not connect to the database to be published. Check the database connection settings (server IP, database name, user, and password) and make sure you’ve installed the Python package for the DBMS you’re using (see the Libs test page in Utilities). <biocase:diagnostic severity="INFO">Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06</biocase:diagnostic><biocase:diagnostic severity="INFO">Load CMFile 'C:\Workspace\bps2\config\datasources\flora\cmf_ABCD_2.06.xml'</biocase:diagnostic><biocase:diagnostic severity="ERROR">The CMF DB mapping file http://www.tdwg.org/schemas/abcd/2.06 was corrupt or could not be interpreted.</biocase:diagnostic> The requested schema (identified by the namespace http://www.tdwg.org/schemas/abcd/2.06) is not supported by that service. Create a mapping and map elements. <biocase:diagnostic severity="ERROR">The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts</biocase:diagnostic> Obvious - happens if you try to use a concept that is not mapped in a scan request or a filter. <biocase:diagnostic severity="ERROR">A SQL statement produced an error: SELECT unit.id, metadata.supplier_organisation, metadata.supplier_email, …, unit.febnr, unit.nameautor, unit.url_gbild, unit.datum1, unit.sammler, unit.fundort, unit.hoehe, metadata.id, unit.id FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) ORDER BY unit.id That could have a number of reasons: One of the table/column names that were set up in the configuration does not exist (because it was renamed or removed) or the credentials used by the BPS do not have sufficient privileges. Simply copy the SQL statement and execute it manually on the database with a regular database client that will show you the detailed error message returned by the DBMS. <biocase:diagnostic severity="WARNING">The mandatory child element SourceInstitutionID could not be created [0 existing, but 1 required]. The element /DataSets/DataSet/Units/Unit was also dropped.</biocase:diagnostic><biocase:diagnostic severity="WARNING">The mandatory child element Units could not be created [0 existing, but 1 required]. The element /DataSets/DataSet was also dropped.</biocase:diagnostic> One problem caused the next one in this sequence of warnings: The mandatory ABCD element was either not mapped or empty; therefore the respective record (“unit”) was dropped. Because this happened for all units in the datasets, all units were dropped, resulting in an empty Units sub tree. The second warning tells you what happened then: Because the Units element is mandatory for the Dataset element, the whole dataset was dropped. As a result, the response ABCD document was empty. == Debugging a BioCASe Installation == In case the debug output of the web service doesn’t help you to find the problem, or if the BPS behaves not as intended without displaying any error messages, you should have a look at the debug logs written by the BPS. To turn on the Debug logs, go to the System Administration page and set Debug logs to True: [[File:instConfigTool.png|500px]] Redo the step that didn’t work and open the log folder of your BioCASe installation. There are several global log files (pywrapper_request.log, pywrapper_error.log, webapp_debug.log, webapp_error.log) and a debug files per data source (e.g. debug_flora.log). Open these files and see if they can give you a clue to the problem. If you still can’t get it running, contact the __BioCASe Team__. 8d2a56a8bb5a2396403b800f1ea7eef1108a59cc 112 111 2011-06-30T15:48:48Z JoergHoletschek 3 /* Sample Errors and Warnings in the Debug Output */ wikitext text/x-wiki This tutorial explains how to debug a BioCASe web service and a BioCASe installation in general. If you follow the instructions and don’t manage to get your service running, please contact the __BioCASe Team__. == Debugging a BioCASe Web Service == Debugging a BioCASe web service can be done with the manual Query Form. It allows you to load a BioCASe request (Search/Scan) from a template, edit the parameters of the request (e.g. the filter) and will show you the response document, which holds the requested ABCD records and useful debug information. === Opening the Query Form === There are several ways to open the Query Form. On the entry page of a datasource, <small>TEST and DEBUG</small> will open the Query Form: [[File:debugDatasourceEntryPage.png]] In the data source configuration tool, the link to the according query form can be always found in the link list at the top of the page. Moreover, in the mapping editor, the button <small>Test mapping!</small> will open the Query Form in a separate tab: [[File:abcdMappingEditorEmptyTop.png]] === Basic Usage === On top of the Query Form you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ''flora'' being the name of the datasource. Make sure this is the correct access point of the web service you want to debug! [[File:debugQueryformFull.png]] The usage of the Query Form is very straight forward: * Click on one of the links below the text box to load a template, for example for an ABCD2 Scan or ABCD2 Search; * Edit the request parameters; * Click on <small>Submit</small> to get the response document. The section <small>BioCASe protocol filter operators</small> lists the operators that can be used for creating complex filters that combine several filter criteria. More about creating filters can be found in the BioCASe Protocol documentation at http://www.biocase.org/products/protocols/index.shtml. Clicking on the <small>Show</small> link in the <small>Capabilities</small> section will show the Capabilities response in a frame of the Query Form. This is useful, since it shows the namespaces for the schemas supported by the web service (which is required in the request text) and XPaths of mapped ABCD elements (required for creating filters on these elements). By using Copy & Paste you can spare the trouble of entering these monstrous strings of characters. === Changing the Debug Level for a Data Source === Debug info can be set to one of four levels for a BioCASe data source: * <strong>Error</strong> is the highest level will only output errors that prevent the web server from answering a request. * <strong>Warning</strong> will output errors that keep BioCASe from answering a request and warnings that don’t prevent the fulfilling of the request, but should be analysed carefully. For example, they might tell you that records have been dropped because mandatory ABCD elements have not been mapped or were empty for certain records. * <strong>Info</strong> is the default and will add some more information about how the Provider Software processes the request. * <strong>Debug</strong> is the most detailed level and will produce very verbose debug output. It adds very comprehensive information about how the ABCD documents are constructed and is suitable only for experienced users. For most cases, keeping the default ''Info'' level should be sufficient for testing and debugging a web service. For production use, you should set it to ''Error'' later. You can do this in the configuration of a datasource under Settings: [[File:debugDatasourceSettings.png]] == Analyzing the Response Document == Upon clicking <small>Submit</small>, the Query Form will send the request to the web service and wait for the response. It might take some time – up to minutes – until the response document shows up. Amongst others, this depends on * the complexity of the query (especially the filters), * the size of the dataset returned, * the debug level set (“debug” results in very exhaustive debug output), * the performance of the servers running BioCASe and the database, * the network bandwidth. The response document consists of three sections (a full response document can be found [[CommonABCD2Concepts|here]]): [[File:debugResponseDocumentOverview.png]] === Header === This section contains information about the BioCASe installation – operating system, DBMS used, Python/BioCASe version etc. It is intended for a consumer application and won’t be very helpful for debugging: <pre> <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> </pre> === Content === This section makes up the payload of the response document, depending on the type of request: <strong>Capabilities:</strong> Lists all schemas supported by the web service. For each schema, all mapped concepts (identified by their XPaths) are listed with data type and searchable flag. <strong>Scan:</strong> The document will simply list all distinct values for the concept specified in the request. When done for the scientific name, it would look similar to this: <pre> <biocase:content recordCount="1664" recordDropped="0" recordStart="0" totalSearchHits="1664"> <biocase:scan> <biocase:value>Achillea atrata L.</biocase:value> <biocase:value>Achillea millefolium L.</biocase:value> <biocase:value>Achillea nobilis L. ssp. nobilis</biocase:value> <biocase:value>Achillea ptarmica L.</biocase:value> <biocase:value>Acinos alpinus (L.) Moench</biocase:value> <biocase:value>Acinos arvensis (Lam.) Dandy</biocase:value> <biocase:value>Aconitum degenii ssp. paniculatum (Arcang.) Mucher</biocase:value> <biocase:value>Aconitum lycoctonum L. ssp. lycoctonum</biocase:value> <biocase:value>Aconitum napellus L. s.l.</biocase:value> ... </biocase:scan> </biocase:content> </pre> <strong>Search:</strong> For this type of request, the content consists of the ABCD dataset(s). See the the [[CommonABCD2Concepts| ABCD2 example]]. === Debug Output === This section is the most interesting part for debugging. If debug level is set to ''Info'' (which is recommended for debugging), it will document the processing of the request by the web service. The output is easy to understand, for example it will contain the SQL statements sent to the database. For a successful search request it could look similar to this: <pre> <biocase:diagnostics> <biocase:diagnostic severity="INFO"> Datasource wrapper FloraExsiccataBavarica requested </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Reading PSF from C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica \provider_setup_file.xml </biocase:diagnostic> <biocase:diagnostic severity="DEBUG"> PSF: PSF=C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\provider_setup_file.xml, recLimit=100, loglevel=20, user=webuser, database=rbg, dbIP=192.168.2.10, dbms=mysql, encoding=latin_1, schemas={u'http://www.tdwg.org/schemas/abcd/2.06': <biocase.wrapper.psf_handler.SupportedSchema instance at 0x011F5B98>}, tablegraph=GRAPH: graph: unit-metadata, +++ ALIAS2TABLE: {u'unit': u'feb3', u'metadata': u'metadata'} </biocase:diagnostic> <biocase:diagnostic severity="INFO"> BioCASe protocol used. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> HTTP parameter 'query' used for building the request. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT DISTINCT unit.id FROM feb3 AS unit WHERE (unit.nameautor LIKE 'Allium %' ) LIMIT 11' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT unit.id, metadata.supplier_organisation, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (40, 41, 42, 43, 44, 45, 46, 47)) ORDER BY unit.id' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> time to execute request is 0.46599984169 </biocase:diagnostic> </biocase:diagnostics> </pre> == Sample Errors and Warnings in the Debug Output == <pre> <biocase:diagnostic severity="ERROR"> No DB connection could be established. Please verify DB state and connection parameters in your provider_setup_file.xml. </biocase:diagnostic> </pre> Well... just what it says. The Provider Software could not connect to the database to be published. Check the database connection settings (server IP, database name, user, and password) and make sure you’ve installed the Python package for the DBMS you’re using (see the <small>Libs test</small> page in <small>Utilities</small>). <pre> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\flora\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="ERROR"> The CMF DB mapping file http://www.tdwg.org/schemas/abcd/2.06 was corrupt or could not be interpreted. </biocase:diagnostic> </pre> The requested schema (identified by the namespace http://www.tdwg.org/schemas/abcd/2.06) is not supported by that service. Create a mapping and map elements. <pre> <biocase:diagnostic severity="ERROR"> The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result /TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts </biocase:diagnostic> </pre> Obvious - happens if you try to use a concept that is not mapped in a scan request or a filter. <pre> <biocase:diagnostic severity="ERROR"> A SQL statement produced an error: SELECT unit.id, metadata.supplier_organisation, metadata.supplier_email, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) ORDER BY unit.id </pre> That could have a number of reasons: One of the table/column names that were set up in the configuration does not exist (because it was renamed or removed) or the credentials used by the BPS do not have sufficient privileges. Simply copy the SQL statement and execute it manually on the database with a regular database client that will show you the detailed error message returned by the DBMS. <pre> <biocase:diagnostic severity="WARNING"> The mandatory child element SourceInstitutionID could not be created [0 existing, but 1 required]. The element /DataSets/DataSet/Units/Unit was also dropped. </biocase:diagnostic> <biocase:diagnostic severity="WARNING"> The mandatory child element Units could not be created [0 existing, but 1 required]. The element /DataSets/DataSet was also dropped. </biocase:diagnostic> </pre> One problem caused the next one in this sequence of warnings: The mandatory ABCD element ''SourceInstitutionID '' was either not mapped or empty; therefore the respective record (''unit'') was dropped. Because this happened for all units in the dataset, all units were dropped, resulting in an empty ''Units'' sub tree. The second warning tells you what happened then: Because the ''Units'' element is mandatory for the ''Dataset'' element, the whole dataset was dropped. As a result, the response ABCD document was empty. == Debugging a BioCASe Installation == In case the debug output of the web service doesn’t help you to find the problem, or if the BPS behaves not as intended without displaying any error messages, you should have a look at the debug logs written by the BPS. To turn on the Debug logs, go to the System Administration page and set Debug logs to True: [[File:instConfigTool.png|500px]] Redo the step that didn’t work and open the log folder of your BioCASe installation. There are several global log files (pywrapper_request.log, pywrapper_error.log, webapp_debug.log, webapp_error.log) and a debug files per data source (e.g. debug_flora.log). Open these files and see if they can give you a clue to the problem. If you still can’t get it running, contact the __BioCASe Team__. bf055aa247417393603563213c136b372a091a37 113 112 2011-06-30T15:49:52Z JoergHoletschek 3 /* Debugging a BioCASe Installation */ wikitext text/x-wiki This tutorial explains how to debug a BioCASe web service and a BioCASe installation in general. If you follow the instructions and don’t manage to get your service running, please contact the __BioCASe Team__. == Debugging a BioCASe Web Service == Debugging a BioCASe web service can be done with the manual Query Form. It allows you to load a BioCASe request (Search/Scan) from a template, edit the parameters of the request (e.g. the filter) and will show you the response document, which holds the requested ABCD records and useful debug information. === Opening the Query Form === There are several ways to open the Query Form. On the entry page of a datasource, <small>TEST and DEBUG</small> will open the Query Form: [[File:debugDatasourceEntryPage.png]] In the data source configuration tool, the link to the according query form can be always found in the link list at the top of the page. Moreover, in the mapping editor, the button <small>Test mapping!</small> will open the Query Form in a separate tab: [[File:abcdMappingEditorEmptyTop.png]] === Basic Usage === On top of the Query Form you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ''flora'' being the name of the datasource. Make sure this is the correct access point of the web service you want to debug! [[File:debugQueryformFull.png]] The usage of the Query Form is very straight forward: * Click on one of the links below the text box to load a template, for example for an ABCD2 Scan or ABCD2 Search; * Edit the request parameters; * Click on <small>Submit</small> to get the response document. The section <small>BioCASe protocol filter operators</small> lists the operators that can be used for creating complex filters that combine several filter criteria. More about creating filters can be found in the BioCASe Protocol documentation at http://www.biocase.org/products/protocols/index.shtml. Clicking on the <small>Show</small> link in the <small>Capabilities</small> section will show the Capabilities response in a frame of the Query Form. This is useful, since it shows the namespaces for the schemas supported by the web service (which is required in the request text) and XPaths of mapped ABCD elements (required for creating filters on these elements). By using Copy & Paste you can spare the trouble of entering these monstrous strings of characters. === Changing the Debug Level for a Data Source === Debug info can be set to one of four levels for a BioCASe data source: * <strong>Error</strong> is the highest level will only output errors that prevent the web server from answering a request. * <strong>Warning</strong> will output errors that keep BioCASe from answering a request and warnings that don’t prevent the fulfilling of the request, but should be analysed carefully. For example, they might tell you that records have been dropped because mandatory ABCD elements have not been mapped or were empty for certain records. * <strong>Info</strong> is the default and will add some more information about how the Provider Software processes the request. * <strong>Debug</strong> is the most detailed level and will produce very verbose debug output. It adds very comprehensive information about how the ABCD documents are constructed and is suitable only for experienced users. For most cases, keeping the default ''Info'' level should be sufficient for testing and debugging a web service. For production use, you should set it to ''Error'' later. You can do this in the configuration of a datasource under Settings: [[File:debugDatasourceSettings.png]] == Analyzing the Response Document == Upon clicking <small>Submit</small>, the Query Form will send the request to the web service and wait for the response. It might take some time – up to minutes – until the response document shows up. Amongst others, this depends on * the complexity of the query (especially the filters), * the size of the dataset returned, * the debug level set (“debug” results in very exhaustive debug output), * the performance of the servers running BioCASe and the database, * the network bandwidth. The response document consists of three sections (a full response document can be found [[CommonABCD2Concepts|here]]): [[File:debugResponseDocumentOverview.png]] === Header === This section contains information about the BioCASe installation – operating system, DBMS used, Python/BioCASe version etc. It is intended for a consumer application and won’t be very helpful for debugging: <pre> <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> </pre> === Content === This section makes up the payload of the response document, depending on the type of request: <strong>Capabilities:</strong> Lists all schemas supported by the web service. For each schema, all mapped concepts (identified by their XPaths) are listed with data type and searchable flag. <strong>Scan:</strong> The document will simply list all distinct values for the concept specified in the request. When done for the scientific name, it would look similar to this: <pre> <biocase:content recordCount="1664" recordDropped="0" recordStart="0" totalSearchHits="1664"> <biocase:scan> <biocase:value>Achillea atrata L.</biocase:value> <biocase:value>Achillea millefolium L.</biocase:value> <biocase:value>Achillea nobilis L. ssp. nobilis</biocase:value> <biocase:value>Achillea ptarmica L.</biocase:value> <biocase:value>Acinos alpinus (L.) Moench</biocase:value> <biocase:value>Acinos arvensis (Lam.) Dandy</biocase:value> <biocase:value>Aconitum degenii ssp. paniculatum (Arcang.) Mucher</biocase:value> <biocase:value>Aconitum lycoctonum L. ssp. lycoctonum</biocase:value> <biocase:value>Aconitum napellus L. s.l.</biocase:value> ... </biocase:scan> </biocase:content> </pre> <strong>Search:</strong> For this type of request, the content consists of the ABCD dataset(s). See the the [[CommonABCD2Concepts| ABCD2 example]]. === Debug Output === This section is the most interesting part for debugging. If debug level is set to ''Info'' (which is recommended for debugging), it will document the processing of the request by the web service. The output is easy to understand, for example it will contain the SQL statements sent to the database. For a successful search request it could look similar to this: <pre> <biocase:diagnostics> <biocase:diagnostic severity="INFO"> Datasource wrapper FloraExsiccataBavarica requested </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Reading PSF from C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica \provider_setup_file.xml </biocase:diagnostic> <biocase:diagnostic severity="DEBUG"> PSF: PSF=C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\provider_setup_file.xml, recLimit=100, loglevel=20, user=webuser, database=rbg, dbIP=192.168.2.10, dbms=mysql, encoding=latin_1, schemas={u'http://www.tdwg.org/schemas/abcd/2.06': <biocase.wrapper.psf_handler.SupportedSchema instance at 0x011F5B98>}, tablegraph=GRAPH: graph: unit-metadata, +++ ALIAS2TABLE: {u'unit': u'feb3', u'metadata': u'metadata'} </biocase:diagnostic> <biocase:diagnostic severity="INFO"> BioCASe protocol used. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> HTTP parameter 'query' used for building the request. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT DISTINCT unit.id FROM feb3 AS unit WHERE (unit.nameautor LIKE 'Allium %' ) LIMIT 11' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT unit.id, metadata.supplier_organisation, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (40, 41, 42, 43, 44, 45, 46, 47)) ORDER BY unit.id' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> time to execute request is 0.46599984169 </biocase:diagnostic> </biocase:diagnostics> </pre> == Sample Errors and Warnings in the Debug Output == <pre> <biocase:diagnostic severity="ERROR"> No DB connection could be established. Please verify DB state and connection parameters in your provider_setup_file.xml. </biocase:diagnostic> </pre> Well... just what it says. The Provider Software could not connect to the database to be published. Check the database connection settings (server IP, database name, user, and password) and make sure you’ve installed the Python package for the DBMS you’re using (see the <small>Libs test</small> page in <small>Utilities</small>). <pre> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\flora\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="ERROR"> The CMF DB mapping file http://www.tdwg.org/schemas/abcd/2.06 was corrupt or could not be interpreted. </biocase:diagnostic> </pre> The requested schema (identified by the namespace http://www.tdwg.org/schemas/abcd/2.06) is not supported by that service. Create a mapping and map elements. <pre> <biocase:diagnostic severity="ERROR"> The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result /TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts </biocase:diagnostic> </pre> Obvious - happens if you try to use a concept that is not mapped in a scan request or a filter. <pre> <biocase:diagnostic severity="ERROR"> A SQL statement produced an error: SELECT unit.id, metadata.supplier_organisation, metadata.supplier_email, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) ORDER BY unit.id </pre> That could have a number of reasons: One of the table/column names that were set up in the configuration does not exist (because it was renamed or removed) or the credentials used by the BPS do not have sufficient privileges. Simply copy the SQL statement and execute it manually on the database with a regular database client that will show you the detailed error message returned by the DBMS. <pre> <biocase:diagnostic severity="WARNING"> The mandatory child element SourceInstitutionID could not be created [0 existing, but 1 required]. The element /DataSets/DataSet/Units/Unit was also dropped. </biocase:diagnostic> <biocase:diagnostic severity="WARNING"> The mandatory child element Units could not be created [0 existing, but 1 required]. The element /DataSets/DataSet was also dropped. </biocase:diagnostic> </pre> One problem caused the next one in this sequence of warnings: The mandatory ABCD element ''SourceInstitutionID '' was either not mapped or empty; therefore the respective record (''unit'') was dropped. Because this happened for all units in the dataset, all units were dropped, resulting in an empty ''Units'' sub tree. The second warning tells you what happened then: Because the ''Units'' element is mandatory for the ''Dataset'' element, the whole dataset was dropped. As a result, the response ABCD document was empty. == Debugging a BioCASe Installation == In case the debug output of the web service doesn’t help you to find the problem, or if the BPS behaves not as intended without displaying any error messages, you should have a look at the debug logs written by the BPS. To turn on the Debug logs, go to the System Administration page and set <small>Debug logs</small> to <small>True</small>: [[File:instConfigTool.png|500px]] Redo the step that didn’t work and open the ''log'' folder of your BioCASe installation. There are several global log files (pywrapper_request.log, pywrapper_error.log, webapp_debug.log, webapp_error.log) and a debug files per data source (e.g. debug_flora.log). Open these files and see if they can give you a clue to the problem. If you still can’t get it running, contact the __BioCASe Team__. c45de2177c84f43b23b964577ac7acac8da8ef28 114 113 2011-06-30T15:53:14Z JoergHoletschek 3 /* Debugging a BioCASe Installation */ wikitext text/x-wiki This tutorial explains how to debug a BioCASe web service and a BioCASe installation in general. If you follow the instructions and don’t manage to get your service running, please contact the __BioCASe Team__. == Debugging a BioCASe Web Service == Debugging a BioCASe web service can be done with the manual Query Form. It allows you to load a BioCASe request (Search/Scan) from a template, edit the parameters of the request (e.g. the filter) and will show you the response document, which holds the requested ABCD records and useful debug information. === Opening the Query Form === There are several ways to open the Query Form. On the entry page of a datasource, <small>TEST and DEBUG</small> will open the Query Form: [[File:debugDatasourceEntryPage.png]] In the data source configuration tool, the link to the according query form can be always found in the link list at the top of the page. Moreover, in the mapping editor, the button <small>Test mapping!</small> will open the Query Form in a separate tab: [[File:abcdMappingEditorEmptyTop.png]] === Basic Usage === On top of the Query Form you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ''flora'' being the name of the datasource. Make sure this is the correct access point of the web service you want to debug! [[File:debugQueryformFull.png]] The usage of the Query Form is very straight forward: * Click on one of the links below the text box to load a template, for example for an ABCD2 Scan or ABCD2 Search; * Edit the request parameters; * Click on <small>Submit</small> to get the response document. The section <small>BioCASe protocol filter operators</small> lists the operators that can be used for creating complex filters that combine several filter criteria. More about creating filters can be found in the BioCASe Protocol documentation at http://www.biocase.org/products/protocols/index.shtml. Clicking on the <small>Show</small> link in the <small>Capabilities</small> section will show the Capabilities response in a frame of the Query Form. This is useful, since it shows the namespaces for the schemas supported by the web service (which is required in the request text) and XPaths of mapped ABCD elements (required for creating filters on these elements). By using Copy & Paste you can spare the trouble of entering these monstrous strings of characters. === Changing the Debug Level for a Data Source === Debug info can be set to one of four levels for a BioCASe data source: * <strong>Error</strong> is the highest level will only output errors that prevent the web server from answering a request. * <strong>Warning</strong> will output errors that keep BioCASe from answering a request and warnings that don’t prevent the fulfilling of the request, but should be analysed carefully. For example, they might tell you that records have been dropped because mandatory ABCD elements have not been mapped or were empty for certain records. * <strong>Info</strong> is the default and will add some more information about how the Provider Software processes the request. * <strong>Debug</strong> is the most detailed level and will produce very verbose debug output. It adds very comprehensive information about how the ABCD documents are constructed and is suitable only for experienced users. For most cases, keeping the default ''Info'' level should be sufficient for testing and debugging a web service. For production use, you should set it to ''Error'' later. You can do this in the configuration of a datasource under Settings: [[File:debugDatasourceSettings.png]] == Analyzing the Response Document == Upon clicking <small>Submit</small>, the Query Form will send the request to the web service and wait for the response. It might take some time – up to minutes – until the response document shows up. Amongst others, this depends on * the complexity of the query (especially the filters), * the size of the dataset returned, * the debug level set (“debug” results in very exhaustive debug output), * the performance of the servers running BioCASe and the database, * the network bandwidth. The response document consists of three sections (a full response document can be found [[CommonABCD2Concepts|here]]): [[File:debugResponseDocumentOverview.png]] === Header === This section contains information about the BioCASe installation – operating system, DBMS used, Python/BioCASe version etc. It is intended for a consumer application and won’t be very helpful for debugging: <pre> <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> </pre> === Content === This section makes up the payload of the response document, depending on the type of request: <strong>Capabilities:</strong> Lists all schemas supported by the web service. For each schema, all mapped concepts (identified by their XPaths) are listed with data type and searchable flag. <strong>Scan:</strong> The document will simply list all distinct values for the concept specified in the request. When done for the scientific name, it would look similar to this: <pre> <biocase:content recordCount="1664" recordDropped="0" recordStart="0" totalSearchHits="1664"> <biocase:scan> <biocase:value>Achillea atrata L.</biocase:value> <biocase:value>Achillea millefolium L.</biocase:value> <biocase:value>Achillea nobilis L. ssp. nobilis</biocase:value> <biocase:value>Achillea ptarmica L.</biocase:value> <biocase:value>Acinos alpinus (L.) Moench</biocase:value> <biocase:value>Acinos arvensis (Lam.) Dandy</biocase:value> <biocase:value>Aconitum degenii ssp. paniculatum (Arcang.) Mucher</biocase:value> <biocase:value>Aconitum lycoctonum L. ssp. lycoctonum</biocase:value> <biocase:value>Aconitum napellus L. s.l.</biocase:value> ... </biocase:scan> </biocase:content> </pre> <strong>Search:</strong> For this type of request, the content consists of the ABCD dataset(s). See the the [[CommonABCD2Concepts| ABCD2 example]]. === Debug Output === This section is the most interesting part for debugging. If debug level is set to ''Info'' (which is recommended for debugging), it will document the processing of the request by the web service. The output is easy to understand, for example it will contain the SQL statements sent to the database. For a successful search request it could look similar to this: <pre> <biocase:diagnostics> <biocase:diagnostic severity="INFO"> Datasource wrapper FloraExsiccataBavarica requested </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Reading PSF from C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica \provider_setup_file.xml </biocase:diagnostic> <biocase:diagnostic severity="DEBUG"> PSF: PSF=C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\provider_setup_file.xml, recLimit=100, loglevel=20, user=webuser, database=rbg, dbIP=192.168.2.10, dbms=mysql, encoding=latin_1, schemas={u'http://www.tdwg.org/schemas/abcd/2.06': <biocase.wrapper.psf_handler.SupportedSchema instance at 0x011F5B98>}, tablegraph=GRAPH: graph: unit-metadata, +++ ALIAS2TABLE: {u'unit': u'feb3', u'metadata': u'metadata'} </biocase:diagnostic> <biocase:diagnostic severity="INFO"> BioCASe protocol used. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> HTTP parameter 'query' used for building the request. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT DISTINCT unit.id FROM feb3 AS unit WHERE (unit.nameautor LIKE 'Allium %' ) LIMIT 11' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT unit.id, metadata.supplier_organisation, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (40, 41, 42, 43, 44, 45, 46, 47)) ORDER BY unit.id' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> time to execute request is 0.46599984169 </biocase:diagnostic> </biocase:diagnostics> </pre> == Sample Errors and Warnings in the Debug Output == <pre> <biocase:diagnostic severity="ERROR"> No DB connection could be established. Please verify DB state and connection parameters in your provider_setup_file.xml. </biocase:diagnostic> </pre> Well... just what it says. The Provider Software could not connect to the database to be published. Check the database connection settings (server IP, database name, user, and password) and make sure you’ve installed the Python package for the DBMS you’re using (see the <small>Libs test</small> page in <small>Utilities</small>). <pre> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\flora\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="ERROR"> The CMF DB mapping file http://www.tdwg.org/schemas/abcd/2.06 was corrupt or could not be interpreted. </biocase:diagnostic> </pre> The requested schema (identified by the namespace http://www.tdwg.org/schemas/abcd/2.06) is not supported by that service. Create a mapping and map elements. <pre> <biocase:diagnostic severity="ERROR"> The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result /TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts </biocase:diagnostic> </pre> Obvious - happens if you try to use a concept that is not mapped in a scan request or a filter. <pre> <biocase:diagnostic severity="ERROR"> A SQL statement produced an error: SELECT unit.id, metadata.supplier_organisation, metadata.supplier_email, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) ORDER BY unit.id </pre> That could have a number of reasons: One of the table/column names that were set up in the configuration does not exist (because it was renamed or removed) or the credentials used by the BPS do not have sufficient privileges. Simply copy the SQL statement and execute it manually on the database with a regular database client that will show you the detailed error message returned by the DBMS. <pre> <biocase:diagnostic severity="WARNING"> The mandatory child element SourceInstitutionID could not be created [0 existing, but 1 required]. The element /DataSets/DataSet/Units/Unit was also dropped. </biocase:diagnostic> <biocase:diagnostic severity="WARNING"> The mandatory child element Units could not be created [0 existing, but 1 required]. The element /DataSets/DataSet was also dropped. </biocase:diagnostic> </pre> One problem caused the next one in this sequence of warnings: The mandatory ABCD element ''SourceInstitutionID '' was either not mapped or empty; therefore the respective record (''unit'') was dropped. Because this happened for all units in the dataset, all units were dropped, resulting in an empty ''Units'' sub tree. The second warning tells you what happened then: Because the ''Units'' element is mandatory for the ''Dataset'' element, the whole dataset was dropped. As a result, the response ABCD document was empty. == Debugging a BioCASe Installation == In case the debug output of the web service doesn’t help you to find the problem, or if the BPS behaves not as intended without displaying any error messages, you should have a look at the debug logs written by the BPS. To turn on the Debug logs, go to the System Administration page and set <small>Debug logs</small> to <small>True</small>. [[File:instConfigTool.png|500px]] Redo the step that didn’t work and open the ''log'' folder of your BioCASe installation. There are several global log files (pywrapper_request.log, pywrapper_error.log, webapp_debug.log, webapp_error.log) and a debug files per data source (e.g. debug_flora.log). Open these files and see if they can give you a clue to the problem. If you still can’t get it running, contact the __BioCASe Team__. a47d5a756b06b2c06f4c86ed9a4b3295d6c73c10 115 114 2011-06-30T15:55:54Z JoergHoletschek 3 /* Basic Usage */ wikitext text/x-wiki This tutorial explains how to debug a BioCASe web service and a BioCASe installation in general. If you follow the instructions and don’t manage to get your service running, please contact the __BioCASe Team__. == Debugging a BioCASe Web Service == Debugging a BioCASe web service can be done with the manual Query Form. It allows you to load a BioCASe request (Search/Scan) from a template, edit the parameters of the request (e.g. the filter) and will show you the response document, which holds the requested ABCD records and useful debug information. === Opening the Query Form === There are several ways to open the Query Form. On the entry page of a datasource, <small>TEST and DEBUG</small> will open the Query Form: [[File:debugDatasourceEntryPage.png]] In the data source configuration tool, the link to the according query form can be always found in the link list at the top of the page. Moreover, in the mapping editor, the button <small>Test mapping!</small> will open the Query Form in a separate tab: [[File:abcdMappingEditorEmptyTop.png]] === Basic Usage === On top of the Query Form you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ''flora'' being the name of the datasource. Make sure this is the correct access point of the web service you want to debug! [[File:debugQueryformFull.png]] The usage of the Query Form is very straightforward: * Click on one of the links below the text box to load a template, for example for an ABCD2 Scan or ABCD2 Search; * Edit the request parameters; * Click on <small>Submit</small> to get the response document. The section <small>BioCASe protocol filter operators</small> lists the operators that can be used for creating complex filters that combine several filter criteria. More about creating filters can be found in the BioCASe Protocol documentation at http://www.biocase.org/products/protocols/index.shtml. Clicking on the <small>Show</small> link in the <small>Capabilities</small> section will show the Capabilities response in a frame of the Query Form. This is useful, since it shows the namespaces for the schemas supported by the web service (which is required in the request text) and XPaths of mapped ABCD elements (required for creating filters on these elements). By using Copy & Paste you can spare the trouble of entering these monstrous strings of characters. === Changing the Debug Level for a Data Source === Debug info can be set to one of four levels for a BioCASe data source: * <strong>Error</strong> is the highest level will only output errors that prevent the web server from answering a request. * <strong>Warning</strong> will output errors that keep BioCASe from answering a request and warnings that don’t prevent the fulfilling of the request, but should be analysed carefully. For example, they might tell you that records have been dropped because mandatory ABCD elements have not been mapped or were empty for certain records. * <strong>Info</strong> is the default and will add some more information about how the Provider Software processes the request. * <strong>Debug</strong> is the most detailed level and will produce very verbose debug output. It adds very comprehensive information about how the ABCD documents are constructed and is suitable only for experienced users. For most cases, keeping the default ''Info'' level should be sufficient for testing and debugging a web service. For production use, you should set it to ''Error'' later. You can do this in the configuration of a datasource under Settings: [[File:debugDatasourceSettings.png]] == Analyzing the Response Document == Upon clicking <small>Submit</small>, the Query Form will send the request to the web service and wait for the response. It might take some time – up to minutes – until the response document shows up. Amongst others, this depends on * the complexity of the query (especially the filters), * the size of the dataset returned, * the debug level set (“debug” results in very exhaustive debug output), * the performance of the servers running BioCASe and the database, * the network bandwidth. The response document consists of three sections (a full response document can be found [[CommonABCD2Concepts|here]]): [[File:debugResponseDocumentOverview.png]] === Header === This section contains information about the BioCASe installation – operating system, DBMS used, Python/BioCASe version etc. It is intended for a consumer application and won’t be very helpful for debugging: <pre> <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> </pre> === Content === This section makes up the payload of the response document, depending on the type of request: <strong>Capabilities:</strong> Lists all schemas supported by the web service. For each schema, all mapped concepts (identified by their XPaths) are listed with data type and searchable flag. <strong>Scan:</strong> The document will simply list all distinct values for the concept specified in the request. When done for the scientific name, it would look similar to this: <pre> <biocase:content recordCount="1664" recordDropped="0" recordStart="0" totalSearchHits="1664"> <biocase:scan> <biocase:value>Achillea atrata L.</biocase:value> <biocase:value>Achillea millefolium L.</biocase:value> <biocase:value>Achillea nobilis L. ssp. nobilis</biocase:value> <biocase:value>Achillea ptarmica L.</biocase:value> <biocase:value>Acinos alpinus (L.) Moench</biocase:value> <biocase:value>Acinos arvensis (Lam.) Dandy</biocase:value> <biocase:value>Aconitum degenii ssp. paniculatum (Arcang.) Mucher</biocase:value> <biocase:value>Aconitum lycoctonum L. ssp. lycoctonum</biocase:value> <biocase:value>Aconitum napellus L. s.l.</biocase:value> ... </biocase:scan> </biocase:content> </pre> <strong>Search:</strong> For this type of request, the content consists of the ABCD dataset(s). See the the [[CommonABCD2Concepts| ABCD2 example]]. === Debug Output === This section is the most interesting part for debugging. If debug level is set to ''Info'' (which is recommended for debugging), it will document the processing of the request by the web service. The output is easy to understand, for example it will contain the SQL statements sent to the database. For a successful search request it could look similar to this: <pre> <biocase:diagnostics> <biocase:diagnostic severity="INFO"> Datasource wrapper FloraExsiccataBavarica requested </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Reading PSF from C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica \provider_setup_file.xml </biocase:diagnostic> <biocase:diagnostic severity="DEBUG"> PSF: PSF=C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\provider_setup_file.xml, recLimit=100, loglevel=20, user=webuser, database=rbg, dbIP=192.168.2.10, dbms=mysql, encoding=latin_1, schemas={u'http://www.tdwg.org/schemas/abcd/2.06': <biocase.wrapper.psf_handler.SupportedSchema instance at 0x011F5B98>}, tablegraph=GRAPH: graph: unit-metadata, +++ ALIAS2TABLE: {u'unit': u'feb3', u'metadata': u'metadata'} </biocase:diagnostic> <biocase:diagnostic severity="INFO"> BioCASe protocol used. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> HTTP parameter 'query' used for building the request. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT DISTINCT unit.id FROM feb3 AS unit WHERE (unit.nameautor LIKE 'Allium %' ) LIMIT 11' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT unit.id, metadata.supplier_organisation, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (40, 41, 42, 43, 44, 45, 46, 47)) ORDER BY unit.id' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> time to execute request is 0.46599984169 </biocase:diagnostic> </biocase:diagnostics> </pre> == Sample Errors and Warnings in the Debug Output == <pre> <biocase:diagnostic severity="ERROR"> No DB connection could be established. Please verify DB state and connection parameters in your provider_setup_file.xml. </biocase:diagnostic> </pre> Well... just what it says. The Provider Software could not connect to the database to be published. Check the database connection settings (server IP, database name, user, and password) and make sure you’ve installed the Python package for the DBMS you’re using (see the <small>Libs test</small> page in <small>Utilities</small>). <pre> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\flora\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="ERROR"> The CMF DB mapping file http://www.tdwg.org/schemas/abcd/2.06 was corrupt or could not be interpreted. </biocase:diagnostic> </pre> The requested schema (identified by the namespace http://www.tdwg.org/schemas/abcd/2.06) is not supported by that service. Create a mapping and map elements. <pre> <biocase:diagnostic severity="ERROR"> The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result /TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts </biocase:diagnostic> </pre> Obvious - happens if you try to use a concept that is not mapped in a scan request or a filter. <pre> <biocase:diagnostic severity="ERROR"> A SQL statement produced an error: SELECT unit.id, metadata.supplier_organisation, metadata.supplier_email, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) ORDER BY unit.id </pre> That could have a number of reasons: One of the table/column names that were set up in the configuration does not exist (because it was renamed or removed) or the credentials used by the BPS do not have sufficient privileges. Simply copy the SQL statement and execute it manually on the database with a regular database client that will show you the detailed error message returned by the DBMS. <pre> <biocase:diagnostic severity="WARNING"> The mandatory child element SourceInstitutionID could not be created [0 existing, but 1 required]. The element /DataSets/DataSet/Units/Unit was also dropped. </biocase:diagnostic> <biocase:diagnostic severity="WARNING"> The mandatory child element Units could not be created [0 existing, but 1 required]. The element /DataSets/DataSet was also dropped. </biocase:diagnostic> </pre> One problem caused the next one in this sequence of warnings: The mandatory ABCD element ''SourceInstitutionID '' was either not mapped or empty; therefore the respective record (''unit'') was dropped. Because this happened for all units in the dataset, all units were dropped, resulting in an empty ''Units'' sub tree. The second warning tells you what happened then: Because the ''Units'' element is mandatory for the ''Dataset'' element, the whole dataset was dropped. As a result, the response ABCD document was empty. == Debugging a BioCASe Installation == In case the debug output of the web service doesn’t help you to find the problem, or if the BPS behaves not as intended without displaying any error messages, you should have a look at the debug logs written by the BPS. To turn on the Debug logs, go to the System Administration page and set <small>Debug logs</small> to <small>True</small>. [[File:instConfigTool.png|500px]] Redo the step that didn’t work and open the ''log'' folder of your BioCASe installation. There are several global log files (pywrapper_request.log, pywrapper_error.log, webapp_debug.log, webapp_error.log) and a debug files per data source (e.g. debug_flora.log). Open these files and see if they can give you a clue to the problem. If you still can’t get it running, contact the __BioCASe Team__. 425f95d619d3d181451a0febe7060ca50ab16f13 116 115 2011-07-04T10:14:07Z JoergHoletschek 3 /* Debug Output */ wikitext text/x-wiki This tutorial explains how to debug a BioCASe web service and a BioCASe installation in general. If you follow the instructions and don’t manage to get your service running, please contact the __BioCASe Team__. == Debugging a BioCASe Web Service == Debugging a BioCASe web service can be done with the manual Query Form. It allows you to load a BioCASe request (Search/Scan) from a template, edit the parameters of the request (e.g. the filter) and will show you the response document, which holds the requested ABCD records and useful debug information. === Opening the Query Form === There are several ways to open the Query Form. On the entry page of a datasource, <small>TEST and DEBUG</small> will open the Query Form: [[File:debugDatasourceEntryPage.png]] In the data source configuration tool, the link to the according query form can be always found in the link list at the top of the page. Moreover, in the mapping editor, the button <small>Test mapping!</small> will open the Query Form in a separate tab: [[File:abcdMappingEditorEmptyTop.png]] === Basic Usage === On top of the Query Form you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ''flora'' being the name of the datasource. Make sure this is the correct access point of the web service you want to debug! [[File:debugQueryformFull.png]] The usage of the Query Form is very straightforward: * Click on one of the links below the text box to load a template, for example for an ABCD2 Scan or ABCD2 Search; * Edit the request parameters; * Click on <small>Submit</small> to get the response document. The section <small>BioCASe protocol filter operators</small> lists the operators that can be used for creating complex filters that combine several filter criteria. More about creating filters can be found in the BioCASe Protocol documentation at http://www.biocase.org/products/protocols/index.shtml. Clicking on the <small>Show</small> link in the <small>Capabilities</small> section will show the Capabilities response in a frame of the Query Form. This is useful, since it shows the namespaces for the schemas supported by the web service (which is required in the request text) and XPaths of mapped ABCD elements (required for creating filters on these elements). By using Copy & Paste you can spare the trouble of entering these monstrous strings of characters. === Changing the Debug Level for a Data Source === Debug info can be set to one of four levels for a BioCASe data source: * <strong>Error</strong> is the highest level will only output errors that prevent the web server from answering a request. * <strong>Warning</strong> will output errors that keep BioCASe from answering a request and warnings that don’t prevent the fulfilling of the request, but should be analysed carefully. For example, they might tell you that records have been dropped because mandatory ABCD elements have not been mapped or were empty for certain records. * <strong>Info</strong> is the default and will add some more information about how the Provider Software processes the request. * <strong>Debug</strong> is the most detailed level and will produce very verbose debug output. It adds very comprehensive information about how the ABCD documents are constructed and is suitable only for experienced users. For most cases, keeping the default ''Info'' level should be sufficient for testing and debugging a web service. For production use, you should set it to ''Error'' later. You can do this in the configuration of a datasource under Settings: [[File:debugDatasourceSettings.png]] == Analyzing the Response Document == Upon clicking <small>Submit</small>, the Query Form will send the request to the web service and wait for the response. It might take some time – up to minutes – until the response document shows up. Amongst others, this depends on * the complexity of the query (especially the filters), * the size of the dataset returned, * the debug level set (“debug” results in very exhaustive debug output), * the performance of the servers running BioCASe and the database, * the network bandwidth. The response document consists of three sections (a full response document can be found [[CommonABCD2Concepts|here]]): [[File:debugResponseDocumentOverview.png]] === Header === This section contains information about the BioCASe installation – operating system, DBMS used, Python/BioCASe version etc. It is intended for a consumer application and won’t be very helpful for debugging: <pre> <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> </pre> === Content === This section makes up the payload of the response document, depending on the type of request: <strong>Capabilities:</strong> Lists all schemas supported by the web service. For each schema, all mapped concepts (identified by their XPaths) are listed with data type and searchable flag. <strong>Scan:</strong> The document will simply list all distinct values for the concept specified in the request. When done for the scientific name, it would look similar to this: <pre> <biocase:content recordCount="1664" recordDropped="0" recordStart="0" totalSearchHits="1664"> <biocase:scan> <biocase:value>Achillea atrata L.</biocase:value> <biocase:value>Achillea millefolium L.</biocase:value> <biocase:value>Achillea nobilis L. ssp. nobilis</biocase:value> <biocase:value>Achillea ptarmica L.</biocase:value> <biocase:value>Acinos alpinus (L.) Moench</biocase:value> <biocase:value>Acinos arvensis (Lam.) Dandy</biocase:value> <biocase:value>Aconitum degenii ssp. paniculatum (Arcang.) Mucher</biocase:value> <biocase:value>Aconitum lycoctonum L. ssp. lycoctonum</biocase:value> <biocase:value>Aconitum napellus L. s.l.</biocase:value> ... </biocase:scan> </biocase:content> </pre> <strong>Search:</strong> For this type of request, the content consists of the ABCD dataset(s). See the the [[CommonABCD2Concepts| ABCD2 example]]. === Diagnostics (Debug Output) === This section is the most interesting part for debugging. If debug level is set to ''Info'' (which is recommended for debugging), it will document the processing of the request by the web service. The output is easy to understand, for example it will contain the SQL statements sent to the database. For a successful search request it could look similar to this: <pre> <biocase:diagnostics> <biocase:diagnostic severity="INFO"> Datasource wrapper FloraExsiccataBavarica requested </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Reading PSF from C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica \provider_setup_file.xml </biocase:diagnostic> <biocase:diagnostic severity="DEBUG"> PSF: PSF=C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\provider_setup_file.xml, recLimit=100, loglevel=20, user=webuser, database=rbg, dbIP=192.168.2.10, dbms=mysql, encoding=latin_1, schemas={u'http://www.tdwg.org/schemas/abcd/2.06': <biocase.wrapper.psf_handler.SupportedSchema instance at 0x011F5B98>}, tablegraph=GRAPH: graph: unit-metadata, +++ ALIAS2TABLE: {u'unit': u'feb3', u'metadata': u'metadata'} </biocase:diagnostic> <biocase:diagnostic severity="INFO"> BioCASe protocol used. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> HTTP parameter 'query' used for building the request. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT DISTINCT unit.id FROM feb3 AS unit WHERE (unit.nameautor LIKE 'Allium %' ) LIMIT 11' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT unit.id, metadata.supplier_organisation, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (40, 41, 42, 43, 44, 45, 46, 47)) ORDER BY unit.id' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> time to execute request is 0.46599984169 </biocase:diagnostic> </biocase:diagnostics> </pre> == Sample Errors and Warnings in the Debug Output == <pre> <biocase:diagnostic severity="ERROR"> No DB connection could be established. Please verify DB state and connection parameters in your provider_setup_file.xml. </biocase:diagnostic> </pre> Well... just what it says. The Provider Software could not connect to the database to be published. Check the database connection settings (server IP, database name, user, and password) and make sure you’ve installed the Python package for the DBMS you’re using (see the <small>Libs test</small> page in <small>Utilities</small>). <pre> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\flora\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="ERROR"> The CMF DB mapping file http://www.tdwg.org/schemas/abcd/2.06 was corrupt or could not be interpreted. </biocase:diagnostic> </pre> The requested schema (identified by the namespace http://www.tdwg.org/schemas/abcd/2.06) is not supported by that service. Create a mapping and map elements. <pre> <biocase:diagnostic severity="ERROR"> The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result /TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts </biocase:diagnostic> </pre> Obvious - happens if you try to use a concept that is not mapped in a scan request or a filter. <pre> <biocase:diagnostic severity="ERROR"> A SQL statement produced an error: SELECT unit.id, metadata.supplier_organisation, metadata.supplier_email, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) ORDER BY unit.id </pre> That could have a number of reasons: One of the table/column names that were set up in the configuration does not exist (because it was renamed or removed) or the credentials used by the BPS do not have sufficient privileges. Simply copy the SQL statement and execute it manually on the database with a regular database client that will show you the detailed error message returned by the DBMS. <pre> <biocase:diagnostic severity="WARNING"> The mandatory child element SourceInstitutionID could not be created [0 existing, but 1 required]. The element /DataSets/DataSet/Units/Unit was also dropped. </biocase:diagnostic> <biocase:diagnostic severity="WARNING"> The mandatory child element Units could not be created [0 existing, but 1 required]. The element /DataSets/DataSet was also dropped. </biocase:diagnostic> </pre> One problem caused the next one in this sequence of warnings: The mandatory ABCD element ''SourceInstitutionID '' was either not mapped or empty; therefore the respective record (''unit'') was dropped. Because this happened for all units in the dataset, all units were dropped, resulting in an empty ''Units'' sub tree. The second warning tells you what happened then: Because the ''Units'' element is mandatory for the ''Dataset'' element, the whole dataset was dropped. As a result, the response ABCD document was empty. == Debugging a BioCASe Installation == In case the debug output of the web service doesn’t help you to find the problem, or if the BPS behaves not as intended without displaying any error messages, you should have a look at the debug logs written by the BPS. To turn on the Debug logs, go to the System Administration page and set <small>Debug logs</small> to <small>True</small>. [[File:instConfigTool.png|500px]] Redo the step that didn’t work and open the ''log'' folder of your BioCASe installation. There are several global log files (pywrapper_request.log, pywrapper_error.log, webapp_debug.log, webapp_error.log) and a debug files per data source (e.g. debug_flora.log). Open these files and see if they can give you a clue to the problem. If you still can’t get it running, contact the __BioCASe Team__. c07a73f3cf3bee1cec0b06296af46a97f8c97e22 136 116 2011-07-18T13:04:03Z JoergHoletschek 3 wikitext text/x-wiki This tutorial explains how to debug a BioCASe web service and a BioCASe installation in general. If you follow the instructions and don’t manage to get your service running, please [[contact the BioCASe team]]. == Debugging a BioCASe Web Service == Debugging a BioCASe web service can be done with the manual Query Form. It allows you to load a BioCASe request (Search/Scan) from a template, edit the parameters of the request (e.g. the filter) and will show you the response document, which holds the requested ABCD records and useful debug information. === Opening the Query Form === There are several ways to open the Query Form. On the entry page of a datasource, <small>TEST and DEBUG</small> will open the Query Form: [[File:debugDatasourceEntryPage.png]] In the data source configuration tool, the link to the according query form can be always found in the link list at the top of the page. Moreover, in the mapping editor, the button <small>Test mapping!</small> will open the Query Form in a separate tab: [[File:abcdMappingEditorEmptyTop.png]] === Basic Usage === On top of the Query Form you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ''flora'' being the name of the datasource. Make sure this is the correct access point of the web service you want to debug! [[File:debugQueryformFull.png]] The usage of the Query Form is very straightforward: * Click on one of the links below the text box to load a template, for example for an ABCD2 Scan or ABCD2 Search; * Edit the request parameters; * Click on <small>Submit</small> to get the response document. The section <small>BioCASe protocol filter operators</small> lists the operators that can be used for creating complex filters that combine several filter criteria. More about creating filters can be found in the BioCASe Protocol documentation at http://www.biocase.org/products/protocols/index.shtml. Clicking on the <small>Show</small> link in the <small>Capabilities</small> section will show the Capabilities response in a frame of the Query Form. This is useful, since it shows the namespaces for the schemas supported by the web service (which is required in the request text) and XPaths of mapped ABCD elements (required for creating filters on these elements). By using Copy & Paste you can spare the trouble of entering these monstrous strings of characters. === Changing the Debug Level for a Data Source === Debug info can be set to one of four levels for a BioCASe data source: * <strong>Error</strong> is the highest level will only output errors that prevent the web server from answering a request. * <strong>Warning</strong> will output errors that keep BioCASe from answering a request and warnings that don’t prevent the fulfilling of the request, but should be analysed carefully. For example, they might tell you that records have been dropped because mandatory ABCD elements have not been mapped or were empty for certain records. * <strong>Info</strong> is the default and will add some more information about how the Provider Software processes the request. * <strong>Debug</strong> is the most detailed level and will produce very verbose debug output. It adds very comprehensive information about how the ABCD documents are constructed and is suitable only for experienced users. For most cases, keeping the default ''Info'' level should be sufficient for testing and debugging a web service. For production use, you should set it to ''Error'' later. You can do this in the configuration of a datasource under Settings: [[File:debugDatasourceSettings.png]] == Analyzing the Response Document == Upon clicking <small>Submit</small>, the Query Form will send the request to the web service and wait for the response. It might take some time – up to minutes – until the response document shows up. Amongst others, this depends on * the complexity of the query (especially the filters), * the size of the dataset returned, * the debug level set (“debug” results in very exhaustive debug output), * the performance of the servers running BioCASe and the database, * the network bandwidth. The response document consists of three sections (a full response document can be found [[CommonABCD2Concepts|here]]): [[File:debugResponseDocumentOverview.png]] === Header === This section contains information about the BioCASe installation – operating system, DBMS used, Python/BioCASe version etc. It is intended for a consumer application and won’t be very helpful for debugging: <pre> <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> </pre> === Content === This section makes up the payload of the response document, depending on the type of request: <strong>Capabilities:</strong> Lists all schemas supported by the web service. For each schema, all mapped concepts (identified by their XPaths) are listed with data type and searchable flag. <strong>Scan:</strong> The document will simply list all distinct values for the concept specified in the request. When done for the scientific name, it would look similar to this: <pre> <biocase:content recordCount="1664" recordDropped="0" recordStart="0" totalSearchHits="1664"> <biocase:scan> <biocase:value>Achillea atrata L.</biocase:value> <biocase:value>Achillea millefolium L.</biocase:value> <biocase:value>Achillea nobilis L. ssp. nobilis</biocase:value> <biocase:value>Achillea ptarmica L.</biocase:value> <biocase:value>Acinos alpinus (L.) Moench</biocase:value> <biocase:value>Acinos arvensis (Lam.) Dandy</biocase:value> <biocase:value>Aconitum degenii ssp. paniculatum (Arcang.) Mucher</biocase:value> <biocase:value>Aconitum lycoctonum L. ssp. lycoctonum</biocase:value> <biocase:value>Aconitum napellus L. s.l.</biocase:value> ... </biocase:scan> </biocase:content> </pre> <strong>Search:</strong> For this type of request, the content consists of the ABCD dataset(s). See the the [[CommonABCD2Concepts| ABCD2 example]]. === Diagnostics (Debug Output) === This section is the most interesting part for debugging. If debug level is set to ''Info'' (which is recommended for debugging), it will document the processing of the request by the web service. The output is easy to understand, for example it will contain the SQL statements sent to the database. For a successful search request it could look similar to this: <pre> <biocase:diagnostics> <biocase:diagnostic severity="INFO"> Datasource wrapper FloraExsiccataBavarica requested </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Reading PSF from C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica \provider_setup_file.xml </biocase:diagnostic> <biocase:diagnostic severity="DEBUG"> PSF: PSF=C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\provider_setup_file.xml, recLimit=100, loglevel=20, user=webuser, database=rbg, dbIP=192.168.2.10, dbms=mysql, encoding=latin_1, schemas={u'http://www.tdwg.org/schemas/abcd/2.06': <biocase.wrapper.psf_handler.SupportedSchema instance at 0x011F5B98>}, tablegraph=GRAPH: graph: unit-metadata, +++ ALIAS2TABLE: {u'unit': u'feb3', u'metadata': u'metadata'} </biocase:diagnostic> <biocase:diagnostic severity="INFO"> BioCASe protocol used. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> HTTP parameter 'query' used for building the request. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT DISTINCT unit.id FROM feb3 AS unit WHERE (unit.nameautor LIKE 'Allium %' ) LIMIT 11' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT unit.id, metadata.supplier_organisation, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (40, 41, 42, 43, 44, 45, 46, 47)) ORDER BY unit.id' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> time to execute request is 0.46599984169 </biocase:diagnostic> </biocase:diagnostics> </pre> == Sample Errors and Warnings in the Debug Output == <pre> <biocase:diagnostic severity="ERROR"> No DB connection could be established. Please verify DB state and connection parameters in your provider_setup_file.xml. </biocase:diagnostic> </pre> Well... just what it says. The Provider Software could not connect to the database to be published. Check the database connection settings (server IP, database name, user, and password) and make sure you’ve installed the Python package for the DBMS you’re using (see the <small>Libs test</small> page in <small>Utilities</small>). <pre> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\flora\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="ERROR"> The CMF DB mapping file http://www.tdwg.org/schemas/abcd/2.06 was corrupt or could not be interpreted. </biocase:diagnostic> </pre> The requested schema (identified by the namespace http://www.tdwg.org/schemas/abcd/2.06) is not supported by that service. Create a mapping and map elements. <pre> <biocase:diagnostic severity="ERROR"> The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result /TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts </biocase:diagnostic> </pre> Obvious - happens if you try to use a concept that is not mapped in a scan request or a filter. <pre> <biocase:diagnostic severity="ERROR"> A SQL statement produced an error: SELECT unit.id, metadata.supplier_organisation, metadata.supplier_email, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) ORDER BY unit.id </pre> That could have a number of reasons: One of the table/column names that were set up in the configuration does not exist (because it was renamed or removed) or the credentials used by the BPS do not have sufficient privileges. Simply copy the SQL statement and execute it manually on the database with a regular database client that will show you the detailed error message returned by the DBMS. <pre> <biocase:diagnostic severity="WARNING"> The mandatory child element SourceInstitutionID could not be created [0 existing, but 1 required]. The element /DataSets/DataSet/Units/Unit was also dropped. </biocase:diagnostic> <biocase:diagnostic severity="WARNING"> The mandatory child element Units could not be created [0 existing, but 1 required]. The element /DataSets/DataSet was also dropped. </biocase:diagnostic> </pre> One problem caused the next one in this sequence of warnings: The mandatory ABCD element ''SourceInstitutionID '' was either not mapped or empty; therefore the respective record (''unit'') was dropped. Because this happened for all units in the dataset, all units were dropped, resulting in an empty ''Units'' sub tree. The second warning tells you what happened then: Because the ''Units'' element is mandatory for the ''Dataset'' element, the whole dataset was dropped. As a result, the response ABCD document was empty. == Debugging a BioCASe Installation == In case the debug output of the web service doesn’t help you to find the problem, or if the BPS behaves not as intended without displaying any error messages, you should have a look at the debug logs written by the BPS. To turn on the Debug logs, go to the System Administration page and set <small>Debug logs</small> to <small>True</small>. [[File:instConfigTool.png|500px]] Redo the step that didn’t work and open the ''log'' folder of your BioCASe installation. There are several global log files (pywrapper_request.log, pywrapper_error.log, webapp_debug.log, webapp_error.log) and a debug files per data source (e.g. debug_flora.log). Open these files and see if they can give you a clue to the problem. If you still can’t get it running, [[contact the BioCASe team]]. 8acb20467998efd85028343eab391064c711fa05 6 45 103 2011-06-30T15:07:29Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 46 107 2011-06-30T15:20:28Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 47 109 2011-06-30T15:23:44Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 0 48 117 2011-07-13T12:05:35Z JoergHoletschek 3 Created page with "= Glossary and Used Abbreviations = ABCD Data Schema (Access to Biological Collection Data): XML data schema used for publishing specimen collections and observational databases..." wikitext text/x-wiki = Glossary and Used Abbreviations = ABCD Data Schema (Access to Biological Collection Data): XML data schema used for publishing specimen collections and observational databases to primary biodiversity networks such as BioCASe and GBIF. The current version of ABCD is 2.06. Apache HTTP Server (“Apache”): Widely used open-source web server software. Apache is available for most operating systems and recommended for running BioCASe. BioCASe Installation: An installation of the BioCASe Provider Software that offers one or several BioCASe Web Services. BioCASe Web Service: A web service offered by a BioCASe installation. A BioCASe web service provides access to the database to be published and can be queried using the BioCASe Protocol. It is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source, for example http://www.foobar.org/biocase/pywrapper.cgi?dsa=Herbar. See ___xxx___ for more information on how to set up a BioCASe Web Service. BioCASe (Biological Collection Access Service) Network: A network of primary biodiversity data providers. Currently, BioCASe offers three different data portals: One on the German flora (search.biocase.de/botany), one for the biodiversity of Europe (search.biocase.org/europe) and one for global occurrence data (search.biocase.org/edit). BioCASE Project (Biological Collection Access Service for Europe): EU-funded project (2002-05) that developed standards and software for publishing primary biodiversity data (specimen collections and observational databases). Within the first years, the results comprised the BioCASe Protocol, the ABCD data schema and the BioCASe Provider Software. BioCASe Protocol: The protocol for querying a BioCASe web service. The protocol defines three types of requests: ‘’Capabilities’’ for introspecting the configuration of a BioCASe web service, ‘’Scan’’ for getting the distinctive values for a concept provided by the web service, and ‘’Search’’ for retrieving the full set of information published for certain records. BioCASe Request: See BioCASe Protocol. BioCASe Technology (Biological Collection Access Service): The suite of standards and software developed by the BioCASE Project and its successors. Currently, the main components are the BioCASe Protocol, the ABCD data schema, BioCASe Provider Software, the SYNTHESYS Cache Generation System and the BioCASe Data Portal Software. BPS: BioCASe Provider Software; also called PyWrapper. Cache: A component (in the BioCASe context, usually a database or several database tables) that temporarily stores data from another source. In most cases, this is done in order to increase performance for information retrieval. Configuration tool (also Config Tool): Set of dialogs used for configuring a BioCASe installation or BioCASe web services. Database View: A virtual table in a database that offers another view on the data stored in the database. A view can combine data from several tables and transform, that is, translate, concatenate or pivot them. Updates of the underlying tables are always reflected in views drawing data from these tables. Datasource: In the BioCASe context, a datasource is database that is to be published as a BioCASe web service. Do not confuse that with an ODBC datasource that defines the connection parameters for connecting to a database through ODBC. DBMS: Database Management System. Debugging: The iterative process of testing, finding and correcting errors. Denormalisation: In a relational database, the information for real world objects are split into several tables, which is called normalisation. Denormalisation is the process of intentionally reversing this step for the sake of faster information retrieval. Firewall: Usually one device or a combination of several devices shielding a local area network (LAN) from the internet. A firewall filters all network traffic between the LAN and the Internet, blocking unwanted protocols (i.e. applications) and ports. In a BioCASe context, servers should be placed in a way that no firewall is located between the web server running the Provider Software and the database server. In case there is a firewall in between, the port used by the database management system (e.g. 3306 for MySQL) must be opened. Global Biodiversity Information Facility (GBIF): International organisation for promoting free and universal access to worldwide primary biodiversity information. One of the main services offered is the GBIF Data Portal (data.gbif.org). Identifier: In a database table, an identifier is a field that names a record in that table. Identifiers are usually unique, that means the identifier is different for each record. Internet Information Services (IIS): Web service platform integrated into Microsoft’s server operating systems (Server 2000, 2003 and 2008); can be used for running the BioCASe Provider Software. Local Query Tool: Component of the BioCASe Provider Software that can be used to access a BioCASe web service. The Query Tool offers a customizable, very simple and no-frills search interface for querying one (!) BioCASe web service. Mapping Editor: Page of the configuration tool for mapping concepts of the published data schema to the source tables/columns in the database. Metadata: Data about data. In the BioCASe context, this is a set of information describing the database published as a whole - title, description, curator’s name, ownership, date of last modification, terms of use etc. Open Database Connectivity (ODBC): Standard Interface for accessing relational database management systems. The connection parameters for connecting to a database through ODBC are defined in an ODBC Datasource. OS: Operating System (Windows, MacOS, Ubuntu, Debian, Ret Hat, Solaris etc.). Python: Platform independent programming language used for the BioCASe Provider Software. PyWrapper: Synonym for the BioCASe Provider Software (BPS), created from ‘’Python Wrapper’’. Sort of ‘’wraps’’ around the database to be published, exposing the data as a BioCASe compliant web service. Query Form: Component of the Provider Software for sending requests to a BioCASe web service and displaying the result documents; usually used for debugging a web service. Schema Mapping: The part of the configuration that tells the Provider Software how to create documents in a specific published data schema from the source database. A BioCASe web service can support several schemas (e.g. for ABCD and DarwinCore) by having several schema mappings. SQL (Structured Query Language): Language for managing data in relational database management systems, e.g. for information retrieval. Subversion: Software versioning and revision control system used for the BioCASe Provider Software. When installing BioCASe, the software can be downloaded directly from the Subversion repository (See ___Installation#Download___). Wrapper: In the BioCASe context a synonym for PyWrapper. 9efc010912163f252bb596f6f3187fb8ffba617b 118 117 2011-07-13T12:09:05Z JoergHoletschek 3 /* Glossary and Used Abbreviations */ wikitext text/x-wiki = Glossary and Used Abbreviations = ;ABCD Data Schema (Access to Biological Collection Data): XML data schema used for publishing specimen collections and observational databases to primary biodiversity networks such as BioCASe and GBIF. The current version of ABCD is 2.06. ;Apache HTTP Server (“Apache”): Widely used open-source web server software. Apache is available for most operating systems and recommended for running BioCASe. ;BioCASe Installation: An installation of the BioCASe Provider Software that offers one or several BioCASe Web Services. ;BioCASe Web Service: A web service offered by a BioCASe installation. A BioCASe web service provides access to the database to be published and can be queried using the BioCASe Protocol. It is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source, for example http://www.foobar.org/biocase/pywrapper.cgi?dsa=Herbar. See ___xxx___ for more information on how to set up a BioCASe Web Service. ;BioCASe (Biological Collection Access Service) Network: A network of primary biodiversity data providers. Currently, BioCASe offers three different data portals: One on the German flora (search.biocase.de/botany), one for the biodiversity of Europe (search.biocase.org/europe) and one for global occurrence data (search.biocase.org/edit). ;BioCASE Project (Biological Collection Access Service for Europe): EU-funded project (2002-05) that developed standards and software for publishing primary biodiversity data (specimen collections and observational databases). Within the first years, the results comprised the BioCASe Protocol, the ABCD data schema and the BioCASe Provider Software. ;BioCASe Protocol: The protocol for querying a BioCASe web service. The protocol defines three types of requests: ‘’Capabilities’’ for introspecting the configuration of a BioCASe web service, ‘’Scan’’ for getting the distinctive values for a concept provided by the web service, and ‘’Search’’ for retrieving the full set of information published for certain records. ;BioCASe Request: See BioCASe Protocol. ;BioCASe Technology (Biological Collection Access Service): The suite of standards and software developed by the BioCASE Project and its successors. Currently, the main components are the BioCASe Protocol, the ABCD data schema, BioCASe Provider Software, the SYNTHESYS Cache Generation System and the BioCASe Data Portal Software. ;BPS: BioCASe Provider Software; also called PyWrapper. ;Cache: A component (in the BioCASe context, usually a database or several database tables) that temporarily stores data from another source. In most cases, this is done in order to increase performance for information retrieval. ;Configuration tool (also Config Tool): Set of dialogs used for configuring a BioCASe installation or BioCASe web services. ;Database View: A virtual table in a database that offers another view on the data stored in the database. A view can combine data from several tables and transform, that is, translate, concatenate or pivot them. Updates of the underlying tables are always reflected in views drawing data from these tables. ;Datasource: In the BioCASe context, a datasource is database that is to be published as a BioCASe web service. Do not confuse that with an ODBC datasource that defines the connection parameters for connecting to a database through ODBC. ;DBMS: Database Management System. ;Debugging: The iterative process of testing, finding and correcting errors. ;Denormalisation: In a relational database, the information for real world objects are split into several tables, which is called normalisation. Denormalisation is the process of intentionally reversing this step for the sake of faster information retrieval. ;Firewall: Usually one device or a combination of several devices shielding a local area network (LAN) from the internet. A firewall filters all network traffic between the LAN and the Internet, blocking unwanted protocols (i.e. applications) and ports. In a BioCASe context, servers should be placed in a way that no firewall is located between the web server running the Provider Software and the database server. In case there is a firewall in between, the port used by the database management system (e.g. 3306 for MySQL) must be opened. ;Global Biodiversity Information Facility (GBIF): International organisation for promoting free and universal access to worldwide primary biodiversity information. One of the main services offered is the GBIF Data Portal (data.gbif.org). ;Identifier: In a database table, an identifier is a field that names a record in that table. Identifiers are usually unique, that means the identifier is different for each record. ;Internet Information Services (IIS): Web service platform integrated into Microsoft’s server operating systems (Server 2000, 2003 and 2008); can be used for running the BioCASe Provider Software. ;Local Query Tool: Component of the BioCASe Provider Software that can be used to access a BioCASe web service. The Query Tool offers a customizable, very simple and no-frills search interface for querying one (!) BioCASe web service. ;Mapping Editor: Page of the configuration tool for mapping concepts of the published data schema to the source tables/columns in the database. ;Metadata: Data about data. In the BioCASe context, this is a set of information describing the database published as a whole - title, description, curator’s name, ownership, date of last modification, terms of use etc. ;Open Database Connectivity (ODBC): Standard Interface for accessing relational database management systems. The connection parameters for connecting to a database through ODBC are defined in an ODBC Datasource. ;OS: Operating System (Windows, MacOS, Ubuntu, Debian, Ret Hat, Solaris etc.). ;Python: Platform independent programming language used for the BioCASe Provider Software. ;PyWrapper: Synonym for the BioCASe Provider Software (BPS), created from ‘’Python Wrapper’’. Sort of ‘’wraps’’ around the database to be published, exposing the data as a BioCASe compliant web service. ;Query Form: Component of the Provider Software for sending requests to a BioCASe web service and displaying the result documents; usually used for debugging a web service. ;Schema Mapping: The part of the configuration that tells the Provider Software how to create documents in a specific published data schema from the source database. A BioCASe web service can support several schemas (e.g. for ABCD and DarwinCore) by having several schema mappings. ;SQL (Structured Query Language): Language for managing data in relational database management systems, e.g. for information retrieval. ;Subversion: Software versioning and revision control system used for the BioCASe Provider Software. When installing BioCASe, the software can be downloaded directly from the Subversion repository (See ___Installation#Download___). ;Wrapper: In the BioCASe context a synonym for PyWrapper. e5e1ba31d205b0e5dbe471cd5bacc235aba4671e 119 118 2011-07-13T12:13:17Z JoergHoletschek 3 /* Glossary and Used Abbreviations */ wikitext text/x-wiki = Glossary and Used Abbreviations = ;ABCD Data Schema (Access to Biological Collection Data): XML data schema used for publishing specimen collections and observational databases to primary biodiversity networks such as BioCASe and GBIF. The current version of ABCD is 2.06. ;Apache HTTP Server (“Apache”): Widely used open-source web server software. Apache is available for most operating systems and recommended for running BioCASe. ;BioCASe Installation: An installation of the BioCASe Provider Software that offers one or several BioCASe Web Services. ;BioCASe Web Service: A web service offered by a BioCASe installation. A BioCASe web service provides access to the database to be published and can be queried using the BioCASe Protocol. It is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source, for example http://www.foobar.org/biocase/pywrapper.cgi?dsa=Herbar. See [[DatasourceSetup]] for more information on how to set up a BioCASe Web Service. ;BioCASe (Biological Collection Access Service) Network: A network of primary biodiversity data providers. Currently, BioCASe offers three different data portals: One on the German flora (search.biocase.de/botany), one for the biodiversity of Europe (search.biocase.org/europe) and one for global occurrence data (search.biocase.org/edit). ;BioCASE Project (Biological Collection Access Service for Europe): EU-funded project (2002-05) that developed standards and software for publishing primary biodiversity data (specimen collections and observational databases). Within the first years, the results comprised the BioCASe Protocol, the ABCD data schema and the BioCASe Provider Software. ;BioCASe Protocol: The protocol for querying a BioCASe web service. The protocol defines three types of requests: ‘’Capabilities’’ for introspecting the configuration of a BioCASe web service, ‘’Scan’’ for getting the distinctive values for a concept provided by the web service, and ‘’Search’’ for retrieving the full set of information published for certain records. ;BioCASe Request: See BioCASe Protocol. ;BioCASe Technology (Biological Collection Access Service): The suite of standards and software developed by the BioCASE Project and its successors. Currently, the main components are the BioCASe Protocol, the ABCD data schema, BioCASe Provider Software, the SYNTHESYS Cache Generation System and the BioCASe Data Portal Software. ;BPS: BioCASe Provider Software; also called PyWrapper. ;Cache: A component (in the BioCASe context, usually a database or several database tables) that temporarily stores data from another source. In most cases, this is done in order to increase performance for information retrieval. ;Configuration tool (also Config Tool): Set of dialogs used for configuring a BioCASe installation or BioCASe web services. ;Database View: A virtual table in a database that offers another view on the data stored in the database. A view can combine data from several tables and transform, that is, translate, concatenate or pivot them. Updates of the underlying tables are always reflected in views drawing data from these tables. ;Datasource: In the BioCASe context, a datasource is database that is to be published as a BioCASe web service. Do not confuse that with an ODBC datasource that defines the connection parameters for connecting to a database through ODBC. ;DBMS: Database Management System. ;Debugging: The iterative process of testing, finding and correcting errors. ;Denormalisation: In a relational database, the information for real world objects are split into several tables, which is called normalisation. Denormalisation is the process of intentionally reversing this step for the sake of faster information retrieval. ;Firewall: Usually one device or a combination of several devices shielding a local area network (LAN) from the internet. A firewall filters all network traffic between the LAN and the Internet, blocking unwanted protocols (i.e. applications) and ports. In a BioCASe context, servers should be placed in a way that no firewall is located between the web server running the Provider Software and the database server. In case there is a firewall in between, the port used by the database management system (e.g. 3306 for MySQL) must be opened. ;Global Biodiversity Information Facility (GBIF): International organisation for promoting free and universal access to worldwide primary biodiversity information. One of the main services offered is the GBIF Data Portal (data.gbif.org). ;Identifier: In a database table, an identifier is a field that names a record in that table. Identifiers are usually unique, that means the identifier is different for each record. ;Internet Information Services (IIS): Web service platform integrated into Microsoft’s server operating systems (Server 2000, 2003 and 2008); can be used for running the BioCASe Provider Software. ;Local Query Tool: Component of the BioCASe Provider Software that can be used to access a BioCASe web service. The Query Tool offers a customizable, very simple and no-frills search interface for querying one (!) BioCASe web service. ;Mapping Editor: Page of the configuration tool for mapping concepts of the published data schema to the source tables/columns in the database. ;Metadata: Data about data. In the BioCASe context, this is a set of information describing the database published as a whole - title, description, curator’s name, ownership, date of last modification, terms of use etc. ;Open Database Connectivity (ODBC): Standard Interface for accessing relational database management systems. The connection parameters for connecting to a database through ODBC are defined in an ODBC Datasource. ;OS: Operating System (Windows, MacOS, Ubuntu, Debian, Ret Hat, Solaris etc.). ;Python: Platform independent programming language used for the BioCASe Provider Software. ;PyWrapper: Synonym for the BioCASe Provider Software (BPS), created from ‘’Python Wrapper’’. Sort of ‘’wraps’’ around the database to be published, exposing the data as a BioCASe compliant web service. ;Query Form: Component of the Provider Software for sending requests to a BioCASe web service and displaying the result documents; usually used for debugging a web service. ;Schema Mapping: The part of the configuration that tells the Provider Software how to create documents in a specific published data schema from the source database. A BioCASe web service can support several schemas (e.g. for ABCD and DarwinCore) by having several schema mappings. ;SQL (Structured Query Language): Language for managing data in relational database management systems, e.g. for information retrieval. ;Subversion: Software versioning and revision control system used for the BioCASe Provider Software. When installing BioCASe, the software can be downloaded directly from the Subversion repository (See [[Installation#Downloading from the Subversion Repository]]). ;Wrapper: In the BioCASe context a synonym for PyWrapper. de46d34e4b7c6e0157e201285b5d6da12d1ad420 120 119 2011-07-13T12:16:56Z JoergHoletschek 3 /* Glossary and Used Abbreviations */ wikitext text/x-wiki = Glossary and Used Abbreviations = ;ABCD Data Schema (Access to Biological Collection Data): XML data schema used for publishing specimen collections and observational databases to primary biodiversity networks such as BioCASe and GBIF. The current version of ABCD is 2.06. ;Apache HTTP Server (“Apache”): Widely used open-source web server software. Apache is available for most operating systems and recommended for running BioCASe. ;BioCASe Installation: An installation of the BioCASe Provider Software that offers one or several BioCASe Web Services. ;BioCASe Web Service: A web service offered by a BioCASe installation. A BioCASe web service provides access to the database to be published and can be queried using the BioCASe Protocol. It is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source, for example http://www.foobar.org/biocase/pywrapper.cgi?dsa=Herbar. See [[DatasourceSetup]] for more information on how to set up a BioCASe Web Service. ;BioCASe (Biological Collection Access Service) Network: A network of primary biodiversity data providers. Currently, BioCASe offers three different data portals: One on the German flora (http://search.biocase.de/botany), one for the biodiversity of Europe (http://search.biocase.org/europe) and one for global occurrence data (http://search.biocase.org/edit). ;BioCASE Project (Biological Collection Access Service for Europe): EU-funded project (2002-05) that developed standards and software for publishing primary biodiversity data (specimen collections and observational databases). Within the first years, the results comprised the BioCASe Protocol, the ABCD data schema and the BioCASe Provider Software. ;BioCASe Protocol: The protocol for querying a BioCASe web service. The protocol defines three types of requests: ''Capabilities'' for introspecting the configuration of a BioCASe web service, ''Scan'' for getting the distinctive values for a concept provided by the web service, and ''Search'' for retrieving the full set of information published for certain records. ;BioCASe Request: See BioCASe Protocol. ;BioCASe Technology (Biological Collection Access Service): The suite of standards and software developed by the BioCASE Project and its successors. Currently, the main components are the BioCASe Protocol, the ABCD data schema, BioCASe Provider Software, the SYNTHESYS Cache Generation System and the BioCASe Data Portal Software. ;BPS: BioCASe Provider Software; also called PyWrapper. ;Cache: A component (in the BioCASe context, usually a database or several database tables) that temporarily stores data from another source. In most cases, this is done in order to increase performance for information retrieval. ;Configuration tool (also Config Tool): Set of dialogs used for configuring a BioCASe installation or BioCASe web services. ;Database View: A virtual table in a database that offers another view on the data stored in the database. A view can combine data from several tables and transform, that is, translate, concatenate or pivot them. Updates of the underlying tables are always reflected in views drawing data from these tables. ;Datasource: In the BioCASe context, a datasource is database that is to be published as a BioCASe web service. Do not confuse that with an ODBC datasource that defines the connection parameters for connecting to a database through ODBC. ;DBMS: Database Management System. ;Debugging: The iterative process of testing, finding and correcting errors. ;Denormalisation: In a relational database, the information for real world objects are split into several tables, which is called normalisation. Denormalisation is the process of intentionally reversing this step for the sake of faster information retrieval. ;Firewall: Usually one device or a combination of several devices shielding a local area network (LAN) from the internet. A firewall filters all network traffic between the LAN and the Internet, blocking unwanted protocols (i.e. applications) and ports. In a BioCASe context, servers should be placed in a way that no firewall is located between the web server running the Provider Software and the database server. In case there is a firewall in between, the port used by the database management system (e.g. 3306 for MySQL) must be opened. ;Global Biodiversity Information Facility (GBIF): International organisation for promoting free and universal access to worldwide primary biodiversity information. One of the main services offered is the GBIF Data Portal (http://data.gbif.org). ;Identifier: In a database table, an identifier is a field that names a record in that table. Identifiers are usually unique, that means the identifier is different for each record. ;Internet Information Services (IIS): Web service platform integrated into Microsoft’s server operating systems (Server 2000, 2003 and 2008); can be used for running the BioCASe Provider Software. ;Local Query Tool: Component of the BioCASe Provider Software that can be used to access a BioCASe web service. The Query Tool offers a customizable, very simple and no-frills search interface for querying '''one''' (!) BioCASe web service. ;Mapping Editor: Page of the configuration tool for mapping concepts of the published data schema to the source tables/columns in the database. ;Metadata: Data about data. In the BioCASe context, this is a set of information describing the database published as a whole - title, description, curator’s name, ownership, date of last modification, terms of use etc. ;Open Database Connectivity (ODBC): Standard Interface for accessing relational database management systems. The connection parameters for connecting to a database through ODBC are defined in an ODBC Datasource. ;OS: Operating System (Windows, MacOS, Ubuntu, Debian, Ret Hat, Solaris etc.). ;Python: Platform independent programming language used for the BioCASe Provider Software. ;PyWrapper: Synonym for the BioCASe Provider Software (BPS), created from "Python Wrapper". Sort of "wraps" around the database to be published, exposing the data as a BioCASe compliant web service. ;Query Form: Component of the Provider Software for sending requests to a BioCASe web service and displaying the result documents; usually used for debugging a web service. ;Schema Mapping: The part of the configuration that tells the Provider Software how to create documents in a specific published data schema from the source database. A BioCASe web service can support several schemas (e.g. for ABCD and DarwinCore) by having several schema mappings. ;SQL (Structured Query Language): Language for managing data in relational database management systems, e.g. for information retrieval. ;Subversion: Software versioning and revision control system used for the BioCASe Provider Software. When installing BioCASe, the software can be downloaded directly from the Subversion repository (See [[Installation#Downloading from the Subversion Repository]]). ;Wrapper: In the BioCASe context a synonym for PyWrapper. d7eab7efec4006c9e8ce3caafb3de01f64b8051e 121 120 2011-07-13T12:19:24Z JoergHoletschek 3 /* Glossary and Used Abbreviations */ wikitext text/x-wiki ;ABCD Data Schema (Access to Biological Collection Data): XML data schema used for publishing specimen collections and observational databases to primary biodiversity networks such as BioCASe and GBIF. The current version of ABCD is 2.06. ;Apache HTTP Server (“Apache”): Widely used open-source web server software. Apache is available for most operating systems and recommended for running BioCASe. ;BioCASe Installation: An installation of the BioCASe Provider Software that offers one or several BioCASe Web Services. ;BioCASe Web Service: A web service offered by a BioCASe installation. A BioCASe web service provides access to the database to be published and can be queried using the BioCASe Protocol. It is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source, for example http://www.foobar.org/biocase/pywrapper.cgi?dsa=Herbar. See [[DatasourceSetup]] for more information on how to set up a BioCASe Web Service. ;BioCASe (Biological Collection Access Service) Network: A network of primary biodiversity data providers. Currently, BioCASe offers three different data portals: One on the German flora (http://search.biocase.de/botany), one for the biodiversity of Europe (http://search.biocase.org/europe) and one for global occurrence data (http://search.biocase.org/edit). ;BioCASE Project (Biological Collection Access Service for Europe): EU-funded project (2002-05) that developed standards and software for publishing primary biodiversity data (specimen collections and observational databases). Within the first years, the results comprised the BioCASe Protocol, the ABCD data schema and the BioCASe Provider Software. ;BioCASe Protocol: The protocol for querying a BioCASe web service. The protocol defines three types of requests: ''Capabilities'' for introspecting the configuration of a BioCASe web service, ''Scan'' for getting the distinctive values for a concept provided by the web service, and ''Search'' for retrieving the full set of information published for certain records. ;BioCASe Request: See BioCASe Protocol. ;BioCASe Technology (Biological Collection Access Service): The suite of standards and software developed by the BioCASE Project and its successors. Currently, the main components are the BioCASe Protocol, the ABCD data schema, BioCASe Provider Software, the SYNTHESYS Cache Generation System and the BioCASe Data Portal Software. ;BPS: BioCASe Provider Software; also called PyWrapper. ;Cache: A component (in the BioCASe context, usually a database or several database tables) that temporarily stores data from another source. In most cases, this is done in order to increase performance for information retrieval. ;Configuration tool (also Config Tool): Set of dialogs used for configuring a BioCASe installation or BioCASe web services. ;Database View: A virtual table in a database that offers another view on the data stored in the database. A view can combine data from several tables and transform, that is, translate, concatenate or pivot them. Updates of the underlying tables are always reflected in views drawing data from these tables. ;Datasource: In the BioCASe context, a datasource is database that is to be published as a BioCASe web service. Do not confuse that with an ODBC datasource that defines the connection parameters for connecting to a database through ODBC. ;DBMS: Database Management System. ;Debugging: The iterative process of testing, finding and correcting errors. ;Denormalisation: In a relational database, the information for real world objects are split into several tables, which is called normalisation. Denormalisation is the process of intentionally reversing this step for the sake of faster information retrieval. ;Firewall: Usually one device or a combination of several devices shielding a local area network (LAN) from the internet. A firewall filters all network traffic between the LAN and the Internet, blocking unwanted protocols (i.e. applications) and ports. In a BioCASe context, servers should be placed in a way that no firewall is located between the web server running the Provider Software and the database server. In case there is a firewall in between, the port used by the database management system (e.g. 3306 for MySQL) must be opened. ;Global Biodiversity Information Facility (GBIF): International organisation for promoting free and universal access to worldwide primary biodiversity information. One of the main services offered is the GBIF Data Portal (http://data.gbif.org). ;Identifier: In a database table, an identifier is a field that names a record in that table. Identifiers are usually unique, that means the identifier is different for each record. ;Internet Information Services (IIS): Web service platform integrated into Microsoft’s server operating systems (Server 2000, 2003 and 2008); can be used for running the BioCASe Provider Software. ;Local Query Tool: Component of the BioCASe Provider Software that can be used to access a BioCASe web service. The Query Tool offers a customizable, very simple and no-frills search interface for querying '''one''' (!) BioCASe web service. ;Mapping Editor: Page of the configuration tool for mapping concepts of the published data schema to the source tables/columns in the database. ;Metadata: Data about data. In the BioCASe context, this is a set of information describing the database published as a whole - title, description, curator’s name, ownership, date of last modification, terms of use etc. ;Open Database Connectivity (ODBC): Standard Interface for accessing relational database management systems. The connection parameters for connecting to a database through ODBC are defined in an ODBC Datasource. ;OS: Operating System (Windows, MacOS, Ubuntu, Debian, Ret Hat, Solaris etc.). ;Python: Platform independent programming language used for the BioCASe Provider Software. ;PyWrapper: Synonym for the BioCASe Provider Software (BPS), created from "Python Wrapper". Sort of "wraps" around the database to be published, exposing the data as a BioCASe compliant web service. ;Query Form: Component of the Provider Software for sending requests to a BioCASe web service and displaying the result documents; usually used for debugging a web service. ;Schema Mapping: The part of the configuration that tells the Provider Software how to create documents in a specific published data schema from the source database. A BioCASe web service can support several schemas (e.g. for ABCD and DarwinCore) by having several schema mappings. ;SQL (Structured Query Language): Language for managing data in relational database management systems, e.g. for information retrieval. ;Subversion: Software versioning and revision control system used for the BioCASe Provider Software. When installing BioCASe, the software can be downloaded directly from the Subversion repository (See [[Installation#Downloading from the Subversion Repository]]). ;Wrapper: In the BioCASe context a synonym for PyWrapper. 5950172cdc334ef8918325520d980418314a1349 0 49 122 2011-07-18T12:38:10Z JoergHoletschek 3 Created page with "==What is the BioCASe Provider Software?== The BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information netwo..." wikitext text/x-wiki ==What is the BioCASe Provider Software?== The BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. The Provider Software is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. The Provider Software is a suite of tools that need to be installed on the data provider’s web server. The core component is the PyWrapper, an XML/CGI database interface written in Python that allows a standardized access to a variety of database management systems and arbitrarily structured databases. Grouped around this are a number of tools for configuring, testing and debugging a BioCASe installation or web service. Even though BioCASe can be used for any conceptual XML data schema, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the __BioCASe network__ and the __Global Biodiversity Information Facility__. ==Requirements for Using the BPS== Before using the BioCASe Provider Software, two requirements need to be checked: • For one thing, the data to be published needs to be stored in an SQL compliant relational database management system (DBMS), with the data model being well documented or at least knowledge about the data model existing on someone’s head. Non-SQL DBMS are not supported. Currently, the BioCASe Provider Software can connect to the following databases: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. • For installing BioCASe, you need to have a web server (Apache or Microsoft IIS) that is permanently connected to the Internet. This can be an institutional server of a hosting service (even though it’s usually a pain to get privileges to install software for the latter). Dial-up connections won’t work. ==Steps== Once you’ve made sure the BioCASe Provider Software is the right choice for you (contact the ___BioCASe Team___ if you feel unsafe with your decision), you need to do the following steps: Think about which information you want to publish to the network. This involves settling the question of ownership of the data and the terms of use, license and usage restrictions for the published data. Also you should consider which information should not be published because they might affect endangered species (options are to block certain occurrences from publication or to blur the exact locality information). Prepare the database for publication as described in the __prepatation tutorial__. This includes the decision on whether to publish the live database or a snapshot, the creation of a table for metadata and, in certain cases, some data transformation. Follow the __Installation Tutorial__ to install the BioCASe Provider Software. Set up a BioCASe Datasource and connect it to your database, then configure the database structure as described in the __Datasource Setup Tutorial__. Follow the ABCD Mapping Tutorial to create a mapping for the schema you want to use. Test and debug your BioCASe web service. Read the __Debugging Tutorial__ to learn how to do this. Once you’re done with setting up and configuring the BioCASe Provider Software, you can register the BioCASe web service with the information network you want to publish your data to, for example with __GBIF__. We strongly recommend to ask the __BioCASe Team__ for checking your web service first as we might be able to foresee problems that are more difficult to resolve after registration. In case you run into difficulties during the installation and setup, you should • have a look at the FAQ which might list the solution for your question, • ask other BioCASe data providers for help, • get in contact with the __BioCASe Team__. Please describe exactly you environment (operating system, DBMS used), the steps that cause the problem and any error messages and hints you get. Screenshot are a big plus here. b4a94b79194e9685f44d04391193d8999d965559 123 122 2011-07-18T12:40:35Z JoergHoletschek 3 /* What is the BioCASe Provider Software? */ wikitext text/x-wiki == What is the BioCASe Provider Software? == The BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. The Provider Software is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. The Provider Software is a suite of tools that need to be installed on the data provider’s web server. The core component is the PyWrapper, an XML/CGI database interface written in Python that allows a standardized access to a variety of database management systems and arbitrarily structured databases. Grouped around this are a number of tools for configuring, testing and debugging a BioCASe installation or web service. Even though BioCASe can be used for any conceptual XML data schema, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the [http://www.biocase.org BioCASe network] and the [http://www.gbif.org Global Biodiversity Information Facility]. ==Requirements for Using the BPS== Before using the BioCASe Provider Software, two requirements need to be checked: • For one thing, the data to be published needs to be stored in an SQL compliant relational database management system (DBMS), with the data model being well documented or at least knowledge about the data model existing on someone’s head. Non-SQL DBMS are not supported. Currently, the BioCASe Provider Software can connect to the following databases: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. • For installing BioCASe, you need to have a web server (Apache or Microsoft IIS) that is permanently connected to the Internet. This can be an institutional server of a hosting service (even though it’s usually a pain to get privileges to install software for the latter). Dial-up connections won’t work. ==Steps== Once you’ve made sure the BioCASe Provider Software is the right choice for you (contact the ___BioCASe Team___ if you feel unsafe with your decision), you need to do the following steps: Think about which information you want to publish to the network. This involves settling the question of ownership of the data and the terms of use, license and usage restrictions for the published data. Also you should consider which information should not be published because they might affect endangered species (options are to block certain occurrences from publication or to blur the exact locality information). Prepare the database for publication as described in the __prepatation tutorial__. This includes the decision on whether to publish the live database or a snapshot, the creation of a table for metadata and, in certain cases, some data transformation. Follow the __Installation Tutorial__ to install the BioCASe Provider Software. Set up a BioCASe Datasource and connect it to your database, then configure the database structure as described in the __Datasource Setup Tutorial__. Follow the ABCD Mapping Tutorial to create a mapping for the schema you want to use. Test and debug your BioCASe web service. Read the __Debugging Tutorial__ to learn how to do this. Once you’re done with setting up and configuring the BioCASe Provider Software, you can register the BioCASe web service with the information network you want to publish your data to, for example with __GBIF__. We strongly recommend to ask the __BioCASe Team__ for checking your web service first as we might be able to foresee problems that are more difficult to resolve after registration. In case you run into difficulties during the installation and setup, you should • have a look at the FAQ which might list the solution for your question, • ask other BioCASe data providers for help, • get in contact with the __BioCASe Team__. Please describe exactly you environment (operating system, DBMS used), the steps that cause the problem and any error messages and hints you get. Screenshot are a big plus here. 353786e8f24b7ea114ee24d0ba792eaa5443e3e1 124 123 2011-07-18T12:41:27Z JoergHoletschek 3 /* Requirements for Using the BPS */ wikitext text/x-wiki == What is the BioCASe Provider Software? == The BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. The Provider Software is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. The Provider Software is a suite of tools that need to be installed on the data provider’s web server. The core component is the PyWrapper, an XML/CGI database interface written in Python that allows a standardized access to a variety of database management systems and arbitrarily structured databases. Grouped around this are a number of tools for configuring, testing and debugging a BioCASe installation or web service. Even though BioCASe can be used for any conceptual XML data schema, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the [http://www.biocase.org BioCASe network] and the [http://www.gbif.org Global Biodiversity Information Facility]. == Requirements for Using the BPS == Before using the BioCASe Provider Software, two requirements need to be checked: * For one thing, the data to be published needs to be stored in an SQL compliant relational database management system (DBMS), with the data model being well documented or at least knowledge about the data model existing on someone’s head. Non-SQL DBMS are not supported. Currently, the BioCASe Provider Software can connect to the following databases: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * For installing BioCASe, you need to have a web server (Apache or Microsoft IIS) that is permanently connected to the Internet. This can be an institutional server of a hosting service (even though it’s usually a pain to get privileges to install software for the latter). Dial-up connections won’t work. ==Steps== Once you’ve made sure the BioCASe Provider Software is the right choice for you (contact the ___BioCASe Team___ if you feel unsafe with your decision), you need to do the following steps: Think about which information you want to publish to the network. This involves settling the question of ownership of the data and the terms of use, license and usage restrictions for the published data. Also you should consider which information should not be published because they might affect endangered species (options are to block certain occurrences from publication or to blur the exact locality information). Prepare the database for publication as described in the __prepatation tutorial__. This includes the decision on whether to publish the live database or a snapshot, the creation of a table for metadata and, in certain cases, some data transformation. Follow the __Installation Tutorial__ to install the BioCASe Provider Software. Set up a BioCASe Datasource and connect it to your database, then configure the database structure as described in the __Datasource Setup Tutorial__. Follow the ABCD Mapping Tutorial to create a mapping for the schema you want to use. Test and debug your BioCASe web service. Read the __Debugging Tutorial__ to learn how to do this. Once you’re done with setting up and configuring the BioCASe Provider Software, you can register the BioCASe web service with the information network you want to publish your data to, for example with __GBIF__. We strongly recommend to ask the __BioCASe Team__ for checking your web service first as we might be able to foresee problems that are more difficult to resolve after registration. In case you run into difficulties during the installation and setup, you should • have a look at the FAQ which might list the solution for your question, • ask other BioCASe data providers for help, • get in contact with the __BioCASe Team__. Please describe exactly you environment (operating system, DBMS used), the steps that cause the problem and any error messages and hints you get. Screenshot are a big plus here. c4c5b927b0779ada2d6593b5707bbf892224225d 125 124 2011-07-18T12:42:55Z JoergHoletschek 3 /* Steps */ wikitext text/x-wiki == What is the BioCASe Provider Software? == The BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. The Provider Software is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. The Provider Software is a suite of tools that need to be installed on the data provider’s web server. The core component is the PyWrapper, an XML/CGI database interface written in Python that allows a standardized access to a variety of database management systems and arbitrarily structured databases. Grouped around this are a number of tools for configuring, testing and debugging a BioCASe installation or web service. Even though BioCASe can be used for any conceptual XML data schema, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the [http://www.biocase.org BioCASe network] and the [http://www.gbif.org Global Biodiversity Information Facility]. == Requirements for Using the BPS == Before using the BioCASe Provider Software, two requirements need to be checked: * For one thing, the data to be published needs to be stored in an SQL compliant relational database management system (DBMS), with the data model being well documented or at least knowledge about the data model existing on someone’s head. Non-SQL DBMS are not supported. Currently, the BioCASe Provider Software can connect to the following databases: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * For installing BioCASe, you need to have a web server (Apache or Microsoft IIS) that is permanently connected to the Internet. This can be an institutional server of a hosting service (even though it’s usually a pain to get privileges to install software for the latter). Dial-up connections won’t work. ==Steps== Once you’ve made sure the BioCASe Provider Software is the right choice for you (contact the ___BioCASe Team___ if you feel unsafe with your decision), you need to do the following steps: * Think about which information you want to publish to the network. This involves settling the question of ownership of the data and the terms of use, license and usage restrictions for the published data. Also you should consider which information should '''not''' be published because they might affect endangered species (options are to block certain occurrences from publication or to blur the exact locality information). * Prepare the database for publication as described in the __prepatation tutorial__. This includes the decision on whether to publish the live database or a snapshot, the creation of a table for metadata and, in certain cases, some data transformation. * Follow the __Installation Tutorial__ to install the BioCASe Provider Software. * Set up a BioCASe Datasource and connect it to your database, then configure the database structure as described in the __Datasource Setup Tutorial__. * Follow the ABCD Mapping Tutorial to create a mapping for the schema you want to use. * Test and debug your BioCASe web service. Read the __Debugging Tutorial__ to learn how to do this. Once you’re done with setting up and configuring the BioCASe Provider Software, you can register the BioCASe web service with the information network you want to publish your data to, for example with __GBIF__. We strongly recommend to ask the __BioCASe Team__ for checking your web service first as we might be able to foresee problems that are more difficult to resolve after registration. In case you run into difficulties during the installation and setup, you should * have a look at the FAQ which might list the solution for your question, * ask other BioCASe data providers for help, * get in contact with the __BioCASe Team__. Please describe exactly you environment (operating system, DBMS used), the steps that cause the problem and any error messages and hints you get. Screenshot are a big plus here. d7bf31cc01d4a10099741f3da6a93dbea677bfd6 126 125 2011-07-18T12:46:29Z JoergHoletschek 3 /* Steps */ wikitext text/x-wiki == What is the BioCASe Provider Software? == The BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. The Provider Software is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. The Provider Software is a suite of tools that need to be installed on the data provider’s web server. The core component is the PyWrapper, an XML/CGI database interface written in Python that allows a standardized access to a variety of database management systems and arbitrarily structured databases. Grouped around this are a number of tools for configuring, testing and debugging a BioCASe installation or web service. Even though BioCASe can be used for any conceptual XML data schema, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the [http://www.biocase.org BioCASe network] and the [http://www.gbif.org Global Biodiversity Information Facility]. == Requirements for Using the BPS == Before using the BioCASe Provider Software, two requirements need to be checked: * For one thing, the data to be published needs to be stored in an SQL compliant relational database management system (DBMS), with the data model being well documented or at least knowledge about the data model existing on someone’s head. Non-SQL DBMS are not supported. Currently, the BioCASe Provider Software can connect to the following databases: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * For installing BioCASe, you need to have a web server (Apache or Microsoft IIS) that is permanently connected to the Internet. This can be an institutional server of a hosting service (even though it’s usually a pain to get privileges to install software for the latter). Dial-up connections won’t work. ==Steps== Once you’ve made sure the BioCASe Provider Software is the right choice for you (contact the ___BioCASe Team___ if you feel unsafe with your decision), you need to do the following steps: * Think about which information you want to publish to the network. This involves settling the question of ownership of the data and the terms of use, license and usage restrictions for the published data. Also you should consider which information should '''not''' be published because they might affect endangered species (options are to block certain occurrences from publication or to blur the exact locality information). * Prepare the database for publication as described in the [[Preparation]] tutorial. This includes the decision on whether to publish the live database or a snapshot, the creation of a table for metadata and, in certain cases, some data transformation. * Follow the [[Installation]] Tutorial to install the BioCASe Provider Software. * Set up a BioCASe Datasource and connect it to your database, then configure the database structure as described in the [[DatasourceSetup]] tutorial. * Follow the [[ABCD2Mapping] tutorial to create a mapping for the schema you want to use. * Test and debug your BioCASe web service. Read the [[Debugging]] tutorial to learn how to do this. Once you’re done with setting up and configuring the BioCASe Provider Software, you can register the BioCASe web service with the information network you want to publish your data to, for example with __GBIF__. We strongly recommend to ask the __BioCASe Team__ for checking your web service first as we might be able to foresee problems that are more difficult to resolve after registration. In case you run into difficulties during the installation and setup, you should * have a look at the FAQ which might list the solution for your question, * ask other BioCASe data providers for help, * get in contact with the __BioCASe Team__. Please describe exactly you environment (operating system, DBMS used), the steps that cause the problem and any error messages and hints you get. Screenshot are a big plus here. 1dfad648fa3c1bd846ddfe2e0fa2df220fd97a82 127 126 2011-07-18T12:49:38Z JoergHoletschek 3 /* Steps */ wikitext text/x-wiki == What is the BioCASe Provider Software? == The BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. The Provider Software is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. The Provider Software is a suite of tools that need to be installed on the data provider’s web server. The core component is the PyWrapper, an XML/CGI database interface written in Python that allows a standardized access to a variety of database management systems and arbitrarily structured databases. Grouped around this are a number of tools for configuring, testing and debugging a BioCASe installation or web service. Even though BioCASe can be used for any conceptual XML data schema, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the [http://www.biocase.org BioCASe network] and the [http://www.gbif.org Global Biodiversity Information Facility]. == Requirements for Using the BPS == Before using the BioCASe Provider Software, two requirements need to be checked: * For one thing, the data to be published needs to be stored in an SQL compliant relational database management system (DBMS), with the data model being well documented or at least knowledge about the data model existing on someone’s head. Non-SQL DBMS are not supported. Currently, the BioCASe Provider Software can connect to the following databases: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * For installing BioCASe, you need to have a web server (Apache or Microsoft IIS) that is permanently connected to the Internet. This can be an institutional server of a hosting service (even though it’s usually a pain to get privileges to install software for the latter). Dial-up connections won’t work. ==Steps== Once you’ve made sure the BioCASe Provider Software is the right choice for you (contact the ___BioCASe Team___ if you feel unsafe with your decision), you need to do the following steps: # Think about which information you want to publish to the network. This involves settling the question of ownership of the data and the terms of use, license and usage restrictions for the published data. Also you should consider which information should '''not''' be published because they might affect endangered species (options are to block certain occurrences from publication or to blur the exact locality information). # Prepare the database for publication as described in the [[Preparation]] tutorial. This includes the decision on whether to publish the live database or a snapshot, the creation of a table for metadata and, in certain cases, some data transformation. # Follow the [[Installation]] Tutorial to install the BioCASe Provider Software. # Set up a BioCASe Datasource and connect it to your database, then configure the database structure as described in the [[DatasourceSetup]] tutorial. # Follow the [[ABCD2Mapping] tutorial to create a mapping for the schema you want to use. # Test and debug your BioCASe web service. Read the [[Debugging]] tutorial to learn how to do this. # Once you’re done with setting up and configuring the BioCASe Provider Software, you can register the BioCASe web service with the information network you want to publish your data to, for example with [[http://www.gbif.org GBIF]]. We strongly recommend to ask the __BioCASe Team__ for checking your web service first as we might be able to foresee problems that are more difficult to resolve after registration. In case you run into difficulties during the installation and setup, you should * have a look at the FAQ which might list the solution for your question, * ask other BioCASe data providers for help, * get in contact with the __BioCASe Team__. Please describe exactly you environment (operating system, DBMS used), the steps that cause the problem and any error messages and hints you get. Screenshot are a big plus here. ffb99cc59dbcd3fad23568b8b4533ec2b4f7e016 128 127 2011-07-18T12:49:55Z JoergHoletschek 3 /* Steps */ wikitext text/x-wiki == What is the BioCASe Provider Software? == The BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. The Provider Software is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. The Provider Software is a suite of tools that need to be installed on the data provider’s web server. The core component is the PyWrapper, an XML/CGI database interface written in Python that allows a standardized access to a variety of database management systems and arbitrarily structured databases. Grouped around this are a number of tools for configuring, testing and debugging a BioCASe installation or web service. Even though BioCASe can be used for any conceptual XML data schema, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the [http://www.biocase.org BioCASe network] and the [http://www.gbif.org Global Biodiversity Information Facility]. == Requirements for Using the BPS == Before using the BioCASe Provider Software, two requirements need to be checked: * For one thing, the data to be published needs to be stored in an SQL compliant relational database management system (DBMS), with the data model being well documented or at least knowledge about the data model existing on someone’s head. Non-SQL DBMS are not supported. Currently, the BioCASe Provider Software can connect to the following databases: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * For installing BioCASe, you need to have a web server (Apache or Microsoft IIS) that is permanently connected to the Internet. This can be an institutional server of a hosting service (even though it’s usually a pain to get privileges to install software for the latter). Dial-up connections won’t work. ==Steps== Once you’ve made sure the BioCASe Provider Software is the right choice for you (contact the ___BioCASe Team___ if you feel unsafe with your decision), you need to do the following steps: # Think about which information you want to publish to the network. This involves settling the question of ownership of the data and the terms of use, license and usage restrictions for the published data. Also you should consider which information should '''not''' be published because they might affect endangered species (options are to block certain occurrences from publication or to blur the exact locality information). # Prepare the database for publication as described in the [[Preparation]] tutorial. This includes the decision on whether to publish the live database or a snapshot, the creation of a table for metadata and, in certain cases, some data transformation. # Follow the [[Installation]] Tutorial to install the BioCASe Provider Software. # Set up a BioCASe Datasource and connect it to your database, then configure the database structure as described in the [[DatasourceSetup]] tutorial. # Follow the [[ABCD2Mapping] tutorial to create a mapping for the schema you want to use. # Test and debug your BioCASe web service. Read the [[Debugging]] tutorial to learn how to do this. # Once you’re done with setting up and configuring the BioCASe Provider Software, you can register the BioCASe web service with the information network you want to publish your data to, for example with [http://www.gbif.org GBIF]. We strongly recommend to ask the __BioCASe Team__ for checking your web service first as we might be able to foresee problems that are more difficult to resolve after registration. In case you run into difficulties during the installation and setup, you should * have a look at the FAQ which might list the solution for your question, * ask other BioCASe data providers for help, * get in contact with the __BioCASe Team__. Please describe exactly you environment (operating system, DBMS used), the steps that cause the problem and any error messages and hints you get. Screenshot are a big plus here. 65b38a1f106bb5806e98e72dd5c521c49a244bf2 132 128 2011-07-18T12:55:23Z JoergHoletschek 3 /* Steps */ wikitext text/x-wiki == What is the BioCASe Provider Software? == The BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. The Provider Software is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. The Provider Software is a suite of tools that need to be installed on the data provider’s web server. The core component is the PyWrapper, an XML/CGI database interface written in Python that allows a standardized access to a variety of database management systems and arbitrarily structured databases. Grouped around this are a number of tools for configuring, testing and debugging a BioCASe installation or web service. Even though BioCASe can be used for any conceptual XML data schema, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the [http://www.biocase.org BioCASe network] and the [http://www.gbif.org Global Biodiversity Information Facility]. == Requirements for Using the BPS == Before using the BioCASe Provider Software, two requirements need to be checked: * For one thing, the data to be published needs to be stored in an SQL compliant relational database management system (DBMS), with the data model being well documented or at least knowledge about the data model existing on someone’s head. Non-SQL DBMS are not supported. Currently, the BioCASe Provider Software can connect to the following databases: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * For installing BioCASe, you need to have a web server (Apache or Microsoft IIS) that is permanently connected to the Internet. This can be an institutional server of a hosting service (even though it’s usually a pain to get privileges to install software for the latter). Dial-up connections won’t work. ==Steps== Once you’ve made sure the BioCASe Provider Software is the right choice for you ([[Contact the BioCASe team]] if you feel unsafe with your decision), you need to do the following steps: # Think about which information you want to publish to the network. This involves settling the question of ownership of the data and the terms of use, license and usage restrictions for the published data. Also you should consider which information should '''not''' be published because they might affect endangered species (options are to block certain occurrences from publication or to blur the exact locality information). # Prepare the database for publication as described in the [[Preparation]] tutorial. This includes the decision on whether to publish the live database or a snapshot, the creation of a table for metadata and, in certain cases, some data transformation. # Follow the [[Installation]] Tutorial to install the BioCASe Provider Software. # Set up a BioCASe Datasource and connect it to your database, then configure the database structure as described in the [[DatasourceSetup]] tutorial. # Follow the [[ABCD2Mapping] tutorial to create a mapping for the schema you want to use. # Test and debug your BioCASe web service. Read the [[Debugging]] tutorial to learn how to do this. # Once you’re done with setting up and configuring the BioCASe Provider Software, you can register the BioCASe web service with the information network you want to publish your data to, for example with [http://www.gbif.org GBIF]. We strongly recommend to ask the __BioCASe Team__ for checking your web service first as we might be able to foresee problems that are more difficult to resolve after registration. In case you run into difficulties during the installation and setup, you should * have a look at the FAQ which might list the solution for your question, * ask other BioCASe data providers for help, * get in contact with the __BioCASe Team__. Please describe exactly you environment (operating system, DBMS used), the steps that cause the problem and any error messages and hints you get. Screenshot are a big plus here. f4ac7b057226a63eda26e21df5736b5bd7d8c222 133 132 2011-07-18T12:56:43Z JoergHoletschek 3 /* Steps */ wikitext text/x-wiki == What is the BioCASe Provider Software? == The BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. The Provider Software is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. The Provider Software is a suite of tools that need to be installed on the data provider’s web server. The core component is the PyWrapper, an XML/CGI database interface written in Python that allows a standardized access to a variety of database management systems and arbitrarily structured databases. Grouped around this are a number of tools for configuring, testing and debugging a BioCASe installation or web service. Even though BioCASe can be used for any conceptual XML data schema, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the [http://www.biocase.org BioCASe network] and the [http://www.gbif.org Global Biodiversity Information Facility]. == Requirements for Using the BPS == Before using the BioCASe Provider Software, two requirements need to be checked: * For one thing, the data to be published needs to be stored in an SQL compliant relational database management system (DBMS), with the data model being well documented or at least knowledge about the data model existing on someone’s head. Non-SQL DBMS are not supported. Currently, the BioCASe Provider Software can connect to the following databases: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * For installing BioCASe, you need to have a web server (Apache or Microsoft IIS) that is permanently connected to the Internet. This can be an institutional server of a hosting service (even though it’s usually a pain to get privileges to install software for the latter). Dial-up connections won’t work. ==Steps== Once you’ve made sure the BioCASe Provider Software is the right choice for you ([[contact the BioCASe team]] if you feel unsafe with your decision), you need to do the following steps: # Think about which information you want to publish to the network. This involves settling the question of ownership of the data and the terms of use, license and usage restrictions for the published data. Also you should consider which information should '''not''' be published because they might affect endangered species (options are to block certain occurrences from publication or to blur the exact locality information). # Prepare the database for publication as described in the [[Preparation]] tutorial. This includes the decision on whether to publish the live database or a snapshot, the creation of a table for metadata and, in certain cases, some data transformation. # Follow the [[Installation]] Tutorial to install the BioCASe Provider Software. # Set up a BioCASe Datasource and connect it to your database, then configure the database structure as described in the [[DatasourceSetup]] tutorial. # Follow the [[ABCD2Mapping] tutorial to create a mapping for the schema you want to use. # Test and debug your BioCASe web service. Read the [[Debugging]] tutorial to learn how to do this. # Once you’re done with setting up and configuring the BioCASe Provider Software, you can register the BioCASe web service with the information network you want to publish your data to, for example with [http://www.gbif.org GBIF]. We strongly recommend to [[contact the BioCASe team]] for checking your web service first as we might be able to foresee problems that are more difficult to resolve after registration. In case you run into difficulties during the installation and setup, you should * have a look at the FAQ which might list the solution for your question, * ask other BioCASe data providers for help, * get in contact with the [[contact the BioCASe team]]. Please describe exactly you environment (operating system, DBMS used), the steps that cause the problem and any error messages and hints you get. Screenshot are a big plus here. a4c2e68c75b26e39d2cc5c37f4dcecdf8c0b1f78 134 133 2011-07-18T12:59:48Z JoergHoletschek 3 /* Steps */ wikitext text/x-wiki == What is the BioCASe Provider Software? == The BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. The Provider Software is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. The Provider Software is a suite of tools that need to be installed on the data provider’s web server. The core component is the PyWrapper, an XML/CGI database interface written in Python that allows a standardized access to a variety of database management systems and arbitrarily structured databases. Grouped around this are a number of tools for configuring, testing and debugging a BioCASe installation or web service. Even though BioCASe can be used for any conceptual XML data schema, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the [http://www.biocase.org BioCASe network] and the [http://www.gbif.org Global Biodiversity Information Facility]. == Requirements for Using the BPS == Before using the BioCASe Provider Software, two requirements need to be checked: * For one thing, the data to be published needs to be stored in an SQL compliant relational database management system (DBMS), with the data model being well documented or at least knowledge about the data model existing on someone’s head. Non-SQL DBMS are not supported. Currently, the BioCASe Provider Software can connect to the following databases: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * For installing BioCASe, you need to have a web server (Apache or Microsoft IIS) that is permanently connected to the Internet. This can be an institutional server of a hosting service (even though it’s usually a pain to get privileges to install software for the latter). Dial-up connections won’t work. ==Steps== Once you’ve made sure the BioCASe Provider Software is the right choice for you ([[contact the BioCASe team]] if you feel unsafe with your decision), you need to do the following steps: # Think about which information you want to publish to the network. This involves settling the question of ownership of the data and the terms of use, license and usage restrictions for the published data. Also you should consider which information should '''not''' be published because they might affect endangered species (options are to block certain occurrences from publication or to blur the exact locality information). # Prepare the database for publication as described in the [[Preparation]] tutorial. This includes the decision on whether to publish the live database or a snapshot, the creation of a table for metadata and, in certain cases, some data transformation. # Follow the [[Installation]] Tutorial to install the BioCASe Provider Software. # Set up a BioCASe Datasource and connect it to your database, then configure the database structure as described in the [[DatasourceSetup]] tutorial. # Follow the [[ABCD2Mapping] tutorial to create a mapping for the schema you want to use. # Test and debug your BioCASe web service. Read the [[Debugging]] tutorial to learn how to do this. # Once you’re done with setting up and configuring the BioCASe Provider Software, you can register the BioCASe web service with the information network you want to publish your data to, for example with [http://www.gbif.org GBIF]. We strongly recommend to [[contact the BioCASe team]] for checking your web service first as we might be able to foresee problems that are more difficult to resolve after registration. In case you run into difficulties during the installation and setup, you should * have a look at the FAQ which might list the solution for your question, * ask other BioCASe data providers for help, * get in contact with the [[contact the BioCASe team|BioCASe team]]. Please describe exactly you environment (operating system, DBMS used), the steps that cause the problem and any error messages and hints you get. Screenshot are a big plus here. 3b6dbe45dac8df9c7a0c0daf65357fa76abed7a2 135 134 2011-07-18T13:00:57Z JoergHoletschek 3 wikitext text/x-wiki == What is the BioCASe Provider Software? == The BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. The Provider Software is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. The Provider Software is a suite of tools that need to be installed on the data provider’s web server. The core component is the PyWrapper, an XML/CGI database interface written in Python that allows a standardized access to a variety of database management systems and arbitrarily structured databases. Grouped around this are a number of tools for configuring, testing and debugging a BioCASe installation or web service. Even though BioCASe can be used for any conceptual XML data schema, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the [http://www.biocase.org BioCASe network] and the [http://www.gbif.org Global Biodiversity Information Facility]. == Requirements for Using the BPS == Before using the BioCASe Provider Software, two requirements need to be checked: * For one thing, the data to be published needs to be stored in an SQL compliant relational database management system (DBMS), with the data model being well documented or at least knowledge about the data model existing on someone’s head. Non-SQL DBMS are not supported. Currently, the BioCASe Provider Software can connect to the following databases: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * For installing BioCASe, you need to have a web server (Apache or Microsoft IIS) that is permanently connected to the Internet. This can be an institutional server of a hosting service (even though it’s usually a pain to get privileges to install software for the latter). Dial-up connections won’t work. == Steps == Once you’ve made sure the BioCASe Provider Software is the right choice for you ([[contact the BioCASe team]] if you feel unsafe with your decision), you need to do the following steps: # Think about which information you want to publish to the network. This involves settling the question of ownership of the data and the terms of use, license and usage restrictions for the published data. Also you should consider which information should '''not''' be published because they might affect endangered species (options are to block certain occurrences from publication or to blur the exact locality information). # Prepare the database for publication as described in the [[Preparation]] tutorial. This includes the decision on whether to publish the live database or a snapshot, the creation of a table for metadata and, in certain cases, some data transformation. # Follow the [[Installation]] Tutorial to install the BioCASe Provider Software. # Set up a BioCASe Datasource and connect it to your database, then configure the database structure as described in the [[DatasourceSetup]] tutorial. # Follow the [[ABCD2Mapping] tutorial to create a mapping for the schema you want to use. # Test and debug your BioCASe web service. Read the [[Debugging]] tutorial to learn how to do this. # Once you’re done with setting up and configuring the BioCASe Provider Software, you can register the BioCASe web service with the information network you want to publish your data to, for example with [http://www.gbif.org GBIF]. We strongly recommend to [[contact the BioCASe team]] for checking your web service first as we might be able to foresee problems that are more difficult to resolve after registration. == Support == In case you run into difficulties during the installation and setup, you should * have a look at the FAQ which might list the solution for your question, * ask other BioCASe data providers for help, * get in contact with the [[contact the BioCASe team|BioCASe team]]. Please describe exactly you environment (operating system, DBMS used), the steps that cause the problem and any error messages and hints you get. Screenshot are a big plus here. b92d3a9e9de88abb1e9c91b07a3b8b6bdfb07d9e 0 17 137 55 2011-07-18T13:05:25Z JoergHoletschek 3 /* Repeatable Elements in ABCD */ wikitext text/x-wiki Before you start publishing your database with BioCASe, you need to consider several issues. In some cases, some preparation of your database for the publication might be necessary. You should read this page of the wiki careful before you start configuring the Provider Software for a given database. == Live-DB versus Snapshot == The easiest way is to connect BioCASe to the same database you’re working with. This would result in any changes you do being visible on the BioCASe access point right away – which is an idea you might not like. Some curators prefer to publish defined versions of their database, for example after they’ve finished reviewing all specimens of a certain genus. If you prefer this, you need to create a second database that holds defined snapshots of your working database and that will be published instead. Snapshots could be generated in a regular interval (e.g., each Friday evening) or upon request. Please contact your IT guys for more help on this. == Access Control == The BioCASe Provider Software will only publish your data to biodiversity networks, meaning it will only read from the database and never do any changes. For security reasons it’s a good idea to create a user for BioCASe that has only read privileges on the database – more precisely on the tables you want to publish. Since the configuration tool will try to load a list of tables and columns from the database to allow easier setup, this user should also be allowed to load meta-information (<small>SHOW TABLES, SHOW COLUMNS</small>). == Metadata == Metadata are information that describe your dataset as a whole - title, description, curator’s name, ownership, date of last modification, terms of use etc. BioCASe can publish a lot of metadata – and you should take the time to provide as much information as you can, since this will increase the usability of your published data. For storing metadata, please create an extra table in your database and fill it in. If you go to the BioCASe folder of your installation, you’ll find the template for this table in ''sampledb/abcdmetadata''. Either use the SQL script to create it in your database, or import it from the Access database (''metadata.mdb''). Per default, all records of your database will make up one dataset, being described by one set of metadata. In case your database holds records of several datasets, you can also group these into several datasets in ABCD by providing different sets of metadata. So when you realize during gathering of meta-information that they’re not the same for all records, this is an indicator that your database must be grouped into several datasets. If all records to be published share the same metadata, the metadata table needs to have just one row. If they don’t, it needs to hold several rows, one set of metadata for each dataset, identified by a unique primary key (1, 2, …). In order to associate each record of your database with the correct set of metadata, you will need to add a foreign key into the base record table that links to the respective entry in the metadata table. Ask your IT guys if you need help on that. == Controlled Denormalisation == As you will see during configuration, the Provider Software is able to join together the different tables of your database’s data model in order to retrieve all information to be published from the database. This works good if the number of tables is within reasonable limits, that is up to approximately 10-15. Data models used by off-the-self collection management system are usually highly normalised and use far more tables. Specify’s data model, for example, has about 70 tables. Configuring such a complex model in BioCASe will be a terrible pain; moreover, in case your database holds a huge number of records, the time required by the Provider Software for answering search request will become too long. Therefore it is advisable in such cases to add some controlled denormalisation to your database, that is to export the data to be published into several cache tables. These tables could be stored in the same database as the source tables or separate in an extra cache database, maybe even on a different server. They could be created in a regular interval (e.g. each night) or upon request by SQL scripts. The figure below shows this denormalisation process for the data model of Specify. On the left side, you see some of Specify’s source tables (only the ones starting with the letters A, B, C and D). With an SQL script the will be reduced to 7 cache tables, one of them being abcd_object. These tables hold only the columns that will be published, drawn from up to 15 source tables. [[image:prepDenormalisation.png|750px]] == Repeatable Elements in ABCD == One of the main features of ABCD (the schema you will probably use to publish your data) is repeatable elements. For example, ABCD can store multiple identifications or several URLs for multimedia objects attached to a specimen record. All these URLs will be stored in the same ABCD element – namely ''MultimediaObject/FileURI''. Another example is the higher taxonomy (all taxa above genus level). Typically these are stored in different columns in databases (see figure below). In ABCD, they will all end up in the same element, ''HigherTaxon/Name''. In order to be able to map the different ranks stored in columns to the same ABCD concept, we need to pivot these columns into rows – each rank kept in a column needs to be turned into a row, with two columns storing taxon name and rank, one column for linking to the according specimen record, plus one newly created primary key: [[File:prepPivoting.png]] Such a transformation can be easily done with a database view. Views do exactly what the name suggests – they provide a different view on the data. They can be used like regular tables, but they merely represent a different representation of another table. Updates of the underlying tables (or insert/deletions) will also affect views created on these tables, since the data is stored only once in the database. Let’s assume you have the higher taxonomy stored in the specimen table like this: [[File:prepEditTable.png]] The SQL statement <pre> CREATE VIEW [dbo].[vwHigherTaxa] AS SELECT 'k_' + [EDIT_ATBI_RecordID] AS id, [EDIT_ATBI_RecordID] AS unit_id, [kingdom] AS name, 'kingdom' AS rank FROM unit_data WHERE [kingdom] IS NOT NULL UNION ALL SELECT 'p_' + [EDIT_ATBI_RecordID], [EDIT_ATBI_RecordID], [phylum], 'phylum'‚ FROM unit_data WHERE [phylum] IS NOT NULL UNION ALL ... </pre> will create a view that meets the requirements described above. Remember that even though it looks like a table, it’s just another representation of the data stored in the table unit_data: [[File:prepEditView.png]] Such pivoting must be done in all cases when several columns need to be mapped to the same (repeatable) ABCD element. If it’s just one value to be mapped, there is no need for that. Commonly used ABCD elements that are repeatable and might ask for such preparatory transformations are * Identification * HigherTaxon * GatheringSite/NamedArea * Metadata/Scope/GeoecologicalTerms * Metadata/Scope/TaxonomicTerms * MultimediaObjects * MeasurementsOrFacts If you have problems preparing your database for publication, please [[contact the BioCASe team]]. 655a1a8199d48b4736c6e5a33edc2f1a16342f90 0 23 138 60 2011-07-18T13:06:48Z JoergHoletschek 3 /* Version 2.5.4 [2010-09-14, withdrawn] */ wikitext text/x-wiki == Version 2 History == ===Version 2.6.1 [upcoming]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.5.5 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. See example [http://ww3.bgbm.org/bps2/ScreenShots?action=AttachFile&do=get&target=config_tool4.gif here] * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool c20ac9b909b1b95d5c8811537d49eff86dcf11c2 0 24 139 73 2011-07-18T13:11:39Z JoergHoletschek 3 wikitext text/x-wiki This tutorial explains in detail how to set up a datasource. == A BioCASe Data Source == Publishing a database with BioCASe means it will be made accessible through a BioCASe web service, which will be used later by the biodiversity network(s) you want to join to access to your data. For example, the GBIF Harvester will send a BioCASe Scan request to that web service to get a list of all scientific names kept in the database, and subsequently several BioCASe Search requests to load all records for these names. For each database you want to publish, you need to set up a BioCASe data source (do not mix that up with ODBC data sources on Windows machines). The resulting BioCASe web service is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source. So if you made your installation available at http://www.foobar.org/biocase during the installation process and set up a data source named Herbar, the URL of the BioCASe web service would be http://www.foobar.org/biocase/pywrapper.cgi?dsa=Herbar (data source names can be case sensitive, depending on your server’s operating system). == Creating a New Data Source == The start page of the BPS will always list the data sources currently configured, that means the web services offered by that BioCASe installation. On a vanilla installation, it will be empty: [[File:instBiocaseStartpage.png|500px]] Creating a new data source can be done in the configuration tool. Choose <small>Config Tool</small> on the start page, then <small>System Administration</small>. Log in with the password you set during the installation process (if you didn’t change that yet, the default is ''ACDC''). In the configuration screen, the sections <small>Data Sources</small> and <small>Recycle Bin</small> can be used to create, remove und un-remove a data source: [[File:dsaCreateDatasourceEmpty.png|500px]] For now, we want to create a new, empty data source. Enter a name – without any spaces – in the text box and click <small>Create DSA</small> (remember that the datasource name is case sensitive on Linux machines). After a second, it will appear in the list of existing data sources with a clickable link. By doing this, you created the first BioCASe web service for your installation. Clicking <small>Remove</small> will uninstall this service again (putting it in the recycle bin, so you can restore it later). Tip: If you select <small>abcdmetadata</small> in the template drop-down list, your data source will be created with an ABCD mapping for all the metadata elements stored in the metadata table mentioned in the [[Preparation]] guide. So if you did use the templates in ''sampledb/abcdmetadata'' to create the metadata table in your database, you can use this template to spare the hassle of mapping this table and start right away with your occurrence data. == Setting up the database connection == Clicking on the data source you’ve just created will take you to the datasource configuration tool. Note that the <small>Schemas</small> section will be empty if you’ve created an empty data source or list two schemas (ABCD 1.2 and 2.06) if you created the datasource based on the <small>abcdmetadata</small> template: [[File:dsaOverview.png|450px]] The link in the <small>Database Connection</small> section will take you to the connection parameters dialog. Fill in the values for your database, press <small>Save</small> and hope for the connection to turn into a nice green <small>OK</small>: '''DBMS:''' Select the entry appropriate for your collection. The BPS can directly connect to database servers (MySQL, Postgres, DB2, SQL Server, and Oracle) and to Desktop-DBMS like Access and FoxPro through ODBC. ODBC is only available on Windows, so if you plan to use it, BioCASe must be installed on a Windows Server. '''Host:''' Enter the IP address of the database server. If you’re connection through ODBC, you must leave this empty. Instead, use Windows’ ODBC Administrator to set up a '''System datasource''' (!); the server or file name of the database will be stored in the ODBC configuration. '''Database:''' Enter the name of the ODBC datasource if you’re using ODBC, the name of the database if you’re directly connecting to a server. '''User/Password:''' Database credentials to be used. As written in the [[Preparation]] guide, you should create a user with just read access for this. '''Encoding:''' The encoding of your database. If you’re not sure, leave the default, you can change it if you run into problems later. If you know your database is using a different encoding than latin-1 (which is typically the default for western languages), maybe UTF-8, select the appropriate entry in the drop-down list. [[File:dsaDBConnection.png|450px]] Once you’ve entered the correct values, press <small>Save</small>. Click <small>Overview</small> in the link list at the top of the page to return to the datasource configuration page. If the connection status doesn’t turn into <small>OK</small> (and doesn’t after you’ve returned to the overview page), please read the __Debug guide__ about how to trace the problem. == Setting up the Database Structure == Clicking on <small>Edit DB Structure</small> on the datasource overview page will show the DB structure setup. Make sure the connection status at the top of the page still is <small>OK</small>. In a first step, you will add all tables/views that hold information you want to publish with BioCASe. You can ignore all database tables storing only data you don’t plan to map to ABCD. Same holds true if you use cache tables (as explained in the [[Preparation]] guide): Only the cache tables need to be set up on the database structure page. In a second step, you will specify the foreign key references between these tables. === Adding Tables/Views with their Primary Keys === To add a table or view, just select it in the drop-down list. The name of the table will be filled in the <small>Alias</small> text box (you will use this alias later in the mapping process), feel free to change it into something that describes the contents of that tables (for example ''specimen'' for the root specimen table). Press <small>Add</small> to insert the table into the list. Before continuing to the next table, select the primary key in the list of attributes under <small>Primary Key Attribute(s)</small>. In case the data type is a string and not an integer, select <small>text</small> from the drop-down list. Press <small>Add</small> to add the primary key. If the primary key consists of several columns, you can add several primary key attributes. Redo this step (adding the tables first, then the primary key(s)) for all tables you want to publish data from. Always remember to set the correct primary key column(s), even if they’re not defined as primary keys in the database. Tables without a primary key cannot be used by the Provider Software and will be removed upon saving! === Specifying Foreign Key References === Once you’ve added all tables, you can define the relations between these tables by defining foreign keys. In order to do this, go to the <small>Foreign Key(s)</small> column of the table that holds a foreign key reference to another table, select the alias of the referenced table and press <small>Add</small>. Select the attribute that stores the foreign key and change the data type if it is not integer, then press <small>Add</small>. You must specify as many attributes for the foreign keys as the primary key of the referenced table has, otherwise the Provider Software won’t be able to create a valid join between the tables. Also, they must be in the same order and have identical data types. Redo this for all tables until you’ve specified all foreign key references. If you’ve installed the Graphviz package, the Provider Software will create a graph visualizing the current database structure. Each table will be visible as a node with the primary key(s) listed. References between tables will be represented by arrows, the foreign key(s) listed in the referring table: [[File:dsaDatabaseStructure_AlgenEngels.png|800px]] === Saving, Editing an Existing Setup === You can change aliases of tables you’ve already added to the list. To do so, go to the <small>Alias</small> text box of that table and type in a different name. Upon exiting the text box, all references to this alias will be updated in the table setup; also the graph will reflect the changes. But caution: Schema mappings use these aliases to bind schema elements to tables and columns, if you change an alias you’ve already used in a schema mapping, you will break this binding! So it is advisable to decide for meaningful alias names and keep them stable once you’ve started the mapping process. Once you’ve added a table/view to the table list, you can’t change the underlying table, since this would invalidate all the primary and foreign key attributes. If you’ve added the wrong table, remove it by using the trash can symbol next to <small>Delete Alias</small> and add the correct table/view. If you’ve finished the table setup and the graph displays the model of your database correctly, save the configuration by pressing <small>Save</small> at the bottom of the page. This is important; navigating away from the page without saving will cause your changes to be lost (even though you will be asked for confirmation)! It is advisable to save the table setup every now and then, for example once you’ve added all tables and primary keys. If you edit an existing table setup and happen to realize you’ve messed it up, you can revert to the state when you’ve pressed Save for the last time by using the <small>Revert</small> button at the bottom. But be careful: All unsaved changes will be lost. '''<u>Important Note:</u>''' The retrieval of tables/view/column names is currently only supported for the following DBMS: Microsoft SQL Server, MySQL, Access, Postgres and FoxPro. If you use a different DBMS, you will see a simple text box instead of a drop-down-list, where you’ll have to type in the name of the table or column you want to add. 75d234449657d86c3a7a350bcb8f37643d529271 144 139 2011-07-18T15:05:15Z JoergHoletschek 3 /* Setting up the database connection */ wikitext text/x-wiki This tutorial explains in detail how to set up a datasource. == A BioCASe Data Source == Publishing a database with BioCASe means it will be made accessible through a BioCASe web service, which will be used later by the biodiversity network(s) you want to join to access to your data. For example, the GBIF Harvester will send a BioCASe Scan request to that web service to get a list of all scientific names kept in the database, and subsequently several BioCASe Search requests to load all records for these names. For each database you want to publish, you need to set up a BioCASe data source (do not mix that up with ODBC data sources on Windows machines). The resulting BioCASe web service is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source. So if you made your installation available at http://www.foobar.org/biocase during the installation process and set up a data source named Herbar, the URL of the BioCASe web service would be http://www.foobar.org/biocase/pywrapper.cgi?dsa=Herbar (data source names can be case sensitive, depending on your server’s operating system). == Creating a New Data Source == The start page of the BPS will always list the data sources currently configured, that means the web services offered by that BioCASe installation. On a vanilla installation, it will be empty: [[File:instBiocaseStartpage.png|500px]] Creating a new data source can be done in the configuration tool. Choose <small>Config Tool</small> on the start page, then <small>System Administration</small>. Log in with the password you set during the installation process (if you didn’t change that yet, the default is ''ACDC''). In the configuration screen, the sections <small>Data Sources</small> and <small>Recycle Bin</small> can be used to create, remove und un-remove a data source: [[File:dsaCreateDatasourceEmpty.png|500px]] For now, we want to create a new, empty data source. Enter a name – without any spaces – in the text box and click <small>Create DSA</small> (remember that the datasource name is case sensitive on Linux machines). After a second, it will appear in the list of existing data sources with a clickable link. By doing this, you created the first BioCASe web service for your installation. Clicking <small>Remove</small> will uninstall this service again (putting it in the recycle bin, so you can restore it later). Tip: If you select <small>abcdmetadata</small> in the template drop-down list, your data source will be created with an ABCD mapping for all the metadata elements stored in the metadata table mentioned in the [[Preparation]] guide. So if you did use the templates in ''sampledb/abcdmetadata'' to create the metadata table in your database, you can use this template to spare the hassle of mapping this table and start right away with your occurrence data. == Setting up the database connection == Clicking on the data source you’ve just created will take you to the datasource configuration tool. Note that the <small>Schemas</small> section will be empty if you’ve created an empty data source or list two schemas (ABCD 1.2 and 2.06) if you created the datasource based on the <small>abcdmetadata</small> template: [[File:dsaOverview.png|450px]] The link in the <small>Database Connection</small> section will take you to the connection parameters dialog. Fill in the values for your database, press <small>Save</small> and hope for the connection to turn into a nice green <small>OK</small>: '''DBMS:''' Select the entry appropriate for your collection. The BPS can directly connect to database servers (MySQL, Postgres, DB2, SQL Server, and Oracle) and to Desktop-DBMS like Access and FoxPro through ODBC. ODBC is only available on Windows, so if you plan to use it, BioCASe must be installed on a Windows Server. '''Host:''' Enter the IP address of the database server. If you’re connection through ODBC, you must leave this empty. Instead, use Windows’ ODBC Administrator to set up a '''System datasource''' (!); the server or file name of the database will be stored in the ODBC configuration. '''Database:''' Enter the name of the ODBC datasource if you’re using ODBC, the name of the database if you’re directly connecting to a server. '''User/Password:''' Database credentials to be used. As written in the [[Preparation]] guide, you should create a user with just read access for this. '''Encoding:''' The encoding of your database. If you’re not sure, leave the default, you can change it if you run into problems later. If you know your database is using a different encoding than latin-1 (which is typically the default for western languages), maybe UTF-8, select the appropriate entry in the drop-down list. [[File:dsaDBConnection.png|450px]] Once you’ve entered the correct values, press <small>Save</small>. Click <small>Overview</small> in the link list at the top of the page to return to the datasource configuration page. If the connection status doesn’t turn into <small>OK</small> (and doesn’t after you’ve returned to the overview page), please read the [[Debugging]] guide about how to trace the problem. == Setting up the Database Structure == Clicking on <small>Edit DB Structure</small> on the datasource overview page will show the DB structure setup. Make sure the connection status at the top of the page still is <small>OK</small>. In a first step, you will add all tables/views that hold information you want to publish with BioCASe. You can ignore all database tables storing only data you don’t plan to map to ABCD. Same holds true if you use cache tables (as explained in the [[Preparation]] guide): Only the cache tables need to be set up on the database structure page. In a second step, you will specify the foreign key references between these tables. === Adding Tables/Views with their Primary Keys === To add a table or view, just select it in the drop-down list. The name of the table will be filled in the <small>Alias</small> text box (you will use this alias later in the mapping process), feel free to change it into something that describes the contents of that tables (for example ''specimen'' for the root specimen table). Press <small>Add</small> to insert the table into the list. Before continuing to the next table, select the primary key in the list of attributes under <small>Primary Key Attribute(s)</small>. In case the data type is a string and not an integer, select <small>text</small> from the drop-down list. Press <small>Add</small> to add the primary key. If the primary key consists of several columns, you can add several primary key attributes. Redo this step (adding the tables first, then the primary key(s)) for all tables you want to publish data from. Always remember to set the correct primary key column(s), even if they’re not defined as primary keys in the database. Tables without a primary key cannot be used by the Provider Software and will be removed upon saving! === Specifying Foreign Key References === Once you’ve added all tables, you can define the relations between these tables by defining foreign keys. In order to do this, go to the <small>Foreign Key(s)</small> column of the table that holds a foreign key reference to another table, select the alias of the referenced table and press <small>Add</small>. Select the attribute that stores the foreign key and change the data type if it is not integer, then press <small>Add</small>. You must specify as many attributes for the foreign keys as the primary key of the referenced table has, otherwise the Provider Software won’t be able to create a valid join between the tables. Also, they must be in the same order and have identical data types. Redo this for all tables until you’ve specified all foreign key references. If you’ve installed the Graphviz package, the Provider Software will create a graph visualizing the current database structure. Each table will be visible as a node with the primary key(s) listed. References between tables will be represented by arrows, the foreign key(s) listed in the referring table: [[File:dsaDatabaseStructure_AlgenEngels.png|800px]] === Saving, Editing an Existing Setup === You can change aliases of tables you’ve already added to the list. To do so, go to the <small>Alias</small> text box of that table and type in a different name. Upon exiting the text box, all references to this alias will be updated in the table setup; also the graph will reflect the changes. But caution: Schema mappings use these aliases to bind schema elements to tables and columns, if you change an alias you’ve already used in a schema mapping, you will break this binding! So it is advisable to decide for meaningful alias names and keep them stable once you’ve started the mapping process. Once you’ve added a table/view to the table list, you can’t change the underlying table, since this would invalidate all the primary and foreign key attributes. If you’ve added the wrong table, remove it by using the trash can symbol next to <small>Delete Alias</small> and add the correct table/view. If you’ve finished the table setup and the graph displays the model of your database correctly, save the configuration by pressing <small>Save</small> at the bottom of the page. This is important; navigating away from the page without saving will cause your changes to be lost (even though you will be asked for confirmation)! It is advisable to save the table setup every now and then, for example once you’ve added all tables and primary keys. If you edit an existing table setup and happen to realize you’ve messed it up, you can revert to the state when you’ve pressed Save for the last time by using the <small>Revert</small> button at the bottom. But be careful: All unsaved changes will be lost. '''<u>Important Note:</u>''' The retrieval of tables/view/column names is currently only supported for the following DBMS: Microsoft SQL Server, MySQL, Access, Postgres and FoxPro. If you use a different DBMS, you will see a simple text box instead of a drop-down-list, where you’ll have to type in the name of the table or column you want to add. 15acc97f70ffb4bf972bbe017db1d34bc77dd675 0 6 140 40 2011-07-18T13:15:29Z JoergHoletschek 3 wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e.g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the ___FAQs___ or get in [[Contact the BioCASe team|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in (in case Python is not found, you’ll have to add the full path name to the Python executable):<pre>python setup.py</pre>Make sure the Python version given at the beginning out the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] All of the questions do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Import configuration from existing installation:</strong> Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.foobar.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.foobar.org/biocase'', for example. After answering these three questions, the script will update your files. At the end of the output it will print the lines you will need later to create the link, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so will compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.foobar.org/biocase''. If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can a pain; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the DBMS used by the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to he library test page that shows the status of installed libraries. On a vanilla BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries ship with the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' in the browser window), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown [[Installation#Linking_BioCASe_and_Your_Web_Server|here]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] 5409392d95b05d9064f6ec4f709d1fb93a1c487b 143 140 2011-07-18T15:04:03Z JoergHoletschek 3 /* Requirements */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e.g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Contact the BioCASe team|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in (in case Python is not found, you’ll have to add the full path name to the Python executable):<pre>python setup.py</pre>Make sure the Python version given at the beginning out the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] All of the questions do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Import configuration from existing installation:</strong> Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.foobar.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.foobar.org/biocase'', for example. After answering these three questions, the script will update your files. At the end of the output it will print the lines you will need later to create the link, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so will compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.foobar.org/biocase''. If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can a pain; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the DBMS used by the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to he library test page that shows the status of installed libraries. On a vanilla BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries ship with the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' in the browser window), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown [[Installation#Linking_BioCASe_and_Your_Web_Server|here]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] e37a6560b283ee9c409c0e135ee6d1d0e4ef7418 0 52 141 2011-07-18T15:01:48Z JoergHoletschek 3 Created page with "FAQ" wikitext text/x-wiki FAQ 03688ba6aa340b87549088aa5739944cb6b1dc73 151 141 2011-07-21T09:54:40Z JoergHoletschek 3 wikitext text/x-wiki == General == What is the BioCASe Provider Software and what can I use it for? See the Beginner’s Guide. When can I use the BioCASe Provider Software? See the requirements in the Beginner’s Guide. I’m not sure whether the Provider Software is the right choice for me. How do I find out? Just contact us and describe your situation and what you’re planning to use BioCASe for. How much is the BioCASe Provider Software? It’s Open Source Software, so you can use it for free. Can I use the Provider Software on my operating system? I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. I cannot meet all of the requirements for installing the BPS. What options do I have? Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. I’ve forgot the password of my BioCASe installation. What should I do? Just go to the config folder of your installation and have a look into the file config.ini. I get a nice pink “A problem occurred in a Python script” error. What’s wrong? Probably something happened we didn’t foresee, or you might just found a bag. Please take a screenshot of the screen and write down what you did to make this happen. Then contact the BioCASe team. I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me. Just contact us; we’ll try our best to help you. I have some recommendations/bug report. How can I file it? You can either leave a message on the comments page or get directly in contact with us. == Installation == I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do? You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. When typing in the URL for my new installation (http://localhost/biocase ), I get an empty page or a “URL not found” error. What’s wrong? If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry (you might need to close and reopen your browser). If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section “Linking BioCASe and Your Web Server” of the Installation guide and follow the instructions. When typing in the URL for my new installation (http://localhost/biocase), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message. You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections “Installing Apache/IIS and Python” and “Running the Setup Script” in the Installation guide). I cannot find the installation package for the DBMS and the Python version I am using. You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. I’ve installed the Python package for my DBMS, but it doesn’t show up in the “Test libs” page. What’s wrong? Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”. This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later. I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page. You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled Graphviz dot binary. On Linux machines it might look similar to the default (/usr/local/bin/dot), on Windows machines it will be probably something like C:\Program Files\Graphviz2.26.3\bin\dot.exe. Press update config to save the path. I want to update my BioCASe installation. How do I do this? First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the config folder of your old installation: config.ini for the system settings and all folders in the datasources directory. == Debugging == I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong? That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other by their primary keys. I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID. That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes. Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press Save to retry connecting. Then see the log files (in the log folder of your biocase installation) for more detailed error messages. The information that can be found in the log files folder is too sparse – what can I do? Turn on debugging in the System Administration. How do I find out if my BioCASe web service works right? Use the Query Form to send a BioCASe request to your web server and check the result document (see the Debugging guide and the Sample ABCD document for more). I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error. The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute. That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ww3.bgbm.org). 0fd6b2779745e20520836c55050633fbe5756627 152 151 2011-07-21T09:57:18Z JoergHoletschek 3 wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the Beginner’s Guide. ===When can I use the BioCASe Provider Software?=== See the requirements in the Beginner’s Guide. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just contact us and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I’ve forgot the password of my BioCASe installation. What should I do?=== Just go to the config folder of your installation and have a look into the file config.ini. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might just found a bag. Please take a screenshot of the screen and write down what you did to make this happen. Then contact the BioCASe team. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just contact us; we’ll try our best to help you. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the comments page or get directly in contact with us. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (http://localhost/biocase ), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry (you might need to close and reopen your browser). If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section “Linking BioCASe and Your Web Server” of the Installation guide and follow the instructions. ===When typing in the URL for my new installation (http://localhost/biocase), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections “Installing Apache/IIS and Python” and “Running the Setup Script” in the Installation guide). ===I cannot find the installation package for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the “Test libs” page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. vI’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled Graphviz dot binary. On Linux machines it might look similar to the default (/usr/local/bin/dot), on Windows machines it will be probably something like C:\Program Files\Graphviz2.26.3\bin\dot.exe. Press update config to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the config folder of your old installation: config.ini for the system settings and all folders in the datasources directory. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press Save to retry connecting. Then see the log files (in the log folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the Debugging guide and the Sample ABCD document for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ww3.bgbm.org). f61c81d21af19f6fa038776356ceb1d4987fc8f4 153 152 2011-07-21T10:04:06Z JoergHoletschek 3 /* General */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===When can I use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just contact us and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I’ve forgot the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might just found a bag. Please take a screenshot of the screen and write down what you did to make this happen. Then [[contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[contact the BioCASe team|contact us]]; we’ll try our best to help you. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the comments page or get directly in [[contact the BioCASe team|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (http://localhost/biocase ), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry (you might need to close and reopen your browser). If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section “Linking BioCASe and Your Web Server” of the Installation guide and follow the instructions. ===When typing in the URL for my new installation (http://localhost/biocase), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections “Installing Apache/IIS and Python” and “Running the Setup Script” in the Installation guide). ===I cannot find the installation package for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the “Test libs” page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. vI’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled Graphviz dot binary. On Linux machines it might look similar to the default (/usr/local/bin/dot), on Windows machines it will be probably something like C:\Program Files\Graphviz2.26.3\bin\dot.exe. Press update config to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the config folder of your old installation: config.ini for the system settings and all folders in the datasources directory. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press Save to retry connecting. Then see the log files (in the log folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the Debugging guide and the Sample ABCD document for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ww3.bgbm.org). ba28d6d61e0581d2cc1a30c465d42cd9c87335f4 154 153 2011-07-21T10:16:28Z JoergHoletschek 3 /* Installation */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===When can I use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just contact us and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I’ve forgot the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might just found a bag. Please take a screenshot of the screen and write down what you did to make this happen. Then [[contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[contact the BioCASe team|contact us]]; we’ll try our best to help you. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the comments page or get directly in [[contact the BioCASe team|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry (you might need to close and reopen your browser). If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I cannot find the installation package for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small>to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press Save to retry connecting. Then see the log files (in the log folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the Debugging guide and the Sample ABCD document for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ww3.bgbm.org). 4e39a11a13982b986590831554e0add0036e1565 155 154 2011-07-21T10:18:51Z JoergHoletschek 3 /* Debugging */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===When can I use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just contact us and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I’ve forgot the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might just found a bag. Please take a screenshot of the screen and write down what you did to make this happen. Then [[contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[contact the BioCASe team|contact us]]; we’ll try our best to help you. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the comments page or get directly in [[contact the BioCASe team|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry (you might need to close and reopen your browser). If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I cannot find the installation package for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small>to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). 53e9446c32cef889eac2ffb2d60b1690b2be9207 156 155 2011-07-21T10:24:51Z JoergHoletschek 3 /* I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page. */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===When can I use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just contact us and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I’ve forgot the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might just found a bag. Please take a screenshot of the screen and write down what you did to make this happen. Then [[contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[contact the BioCASe team|contact us]]; we’ll try our best to help you. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the comments page or get directly in [[contact the BioCASe team|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry (you might need to close and reopen your browser). If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I cannot find the installation package for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). f800470179776acad33b204723fbb47e1ed0aa64 158 156 2011-07-21T10:44:02Z JoergHoletschek 3 /* I have some recommendations/bug report. How can I file it? */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===When can I use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just contact us and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I’ve forgot the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might just found a bag. Please take a screenshot of the screen and write down what you did to make this happen. Then [[contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[contact the BioCASe team|contact us]]; we’ll try our best to help you. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the [[Comments]] page or get directly in [[contact the BioCASe team|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry (you might need to close and reopen your browser). If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I cannot find the installation package for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). 385134242b42a6f12b59635e7c3026386a9cec27 0 29 142 92 2011-07-18T15:03:09Z JoergHoletschek 3 /* Initial Testing of the Data Source */ wikitext text/x-wiki This tutorial will explain how to set up a mapping for the ABCD2.06 data schema. If you intend to map another schema (e.g. one of the ABCD extension ABCD-EFG, ABCD-DNA or HISPID), you still should take the time to read it. Large portions of the process will be identical. == Access to Biological Collection Data Schema (ABCD) == ABCD in its current version 2.06 is the data schema typically used in conjunction with BioCASe. It is a highly complex XML data schema with about 1,000 elements that is able to store almost every piece of information that can be found in a natural history specimen collection or an observation database. The complexity is due to the fact that it can be used for a wide range of collections/databases – for living and preserved specimens, observations and culture collections, for zoological, botanical, bacterial and viral collections, marine or terrestrial, for herbaria, botanic and zoological gardens. For each of these special types, ABCD features special sections where information specific for this type can be stored. Other sections of ABCD will be shared by all these types, for example for gathering site, identifier or metadata. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a list of commonly used ABCD elements on the Wiki page [[CommonABCD2Concepts]]. == Creating an ABCD Mapping for a Data Source == If you created a data source from a template (e.g. ''abcdmetadata''), a schema mapping for ABCD will be already existent. In this case, you can directly jump to the next section. If you created an empty data source, the section <small>Schemas</small> in the data source configuration overview will be empty: [[File:abcdSchemaMappings.png]] In order to create a new schema mapping, select the desired schema from the list and click <small>Create</small>. For ABCD, choose <small>ABCD_2.06.xml</small> (be careful not to hit one of the ABCD extensions, they look quite similar). After clicking <small>Create</small>, you will be directed to the mapping editor, which will allow you to edit the mappings. == The Mapping Editor == The mapping editor allows you to add, remove and edit ABCD mappings. On top you’ll find some summary information, on the bottom you’ll see the structure of the ABCD mapping: [[File:abcdMappingEditorEmpty.png]] First thing we will do is to set the root table, which is the table that holds the records you want to publish (specimens or observations). That table must hold one record per object to be published, identified by a unique identifier (called ''UnitID''). This identifier must not be the primary key, as long as it is unique (if that sentence confuses you, ignore it and just keep in mind that the UnitID must be unique.) Use the drop down box labelled <small>Root table Alias</small> to select the root table of your data model. If your metadata table holds just one record (because all occurrences published share the same metadata) and if it is not linked to the root table by a foreign key, you can specify this table in the drop-down box <small>Static table Alias</small>. The Provider Software will use a natural join to connect the metadata record stored in this table to all occurrences published. If your metadata table is referenced by a foreign key in the root able, you must not declare the metadata table as a static table! So either join the metadata table using a foreign key OR declare it as a static table. When you’ve set the root table (and static table, if you use that), press <small>Save</small>. As with the DB structure setup, this will write your changes to the configuration files, closing the tab without saving will discard your changes! Also, in case you messed something up, you can use the <small>Revert</small> button to restore the state when you’ve saved for the last time. In the mapping editor, the link <small>Overview</small> at the top of the page will take you back to the datasource configuration overview. After creating a mapping, it will appear in the section <small>Schemas</small>. The entry will show the number of mapped elements (0 for now) and the schema namespace. Pressing the trash can symbol will move it to the trash can, which means that your web service does not support this schema anymore; if you did this by mistake, you can restore it by pressing the <small>Restore</small> button. <small>Purge</small> will delete the schema mapping permanently: [[File:abcdSchemaMappings2.png]] You can map several schemas for a given datasource. For example, in addition to the current ABCD2 standard, you could add the deprecated version 1.2 (even though I have no idea why you should do that) or DarwinCore. Just select the appropriate schema in the drop-down list and click <small>Create</small>. Since the namespace of a schema will be used in requests, you cannot create two mappings for the same namespace. Trying this will get you an error message. == Adding Mandatory ABCD Elements == Clicking on a schema in the datasource overview page will take you again to the mapping editor. In the lower half of the page, you will see the basic structure of an ABCD document: [[File:abcdMappingEditorEmptyBottom.png]] <strong>Technical/Content Contact:</strong> These trees hold information on who should be contacted for technical issues or questions concerning the published database, each with name, address, email address and telephone number. <strong>Metadata:</strong> In this sub tree you’ll find all metadata fields, for example title and description of the dataset, ownership information, date last modified, IPR statements etc. <strong>Units:</strong> This part stores the "real" data you want to publish, that means all specimen or observation data. The three elements already shown in the tree are identifiers that make up the so called "Triple ID", which uniquely identifies records within primary biodiversity networks such as BioCASe and GBIF: * ''UnitID'', which is a unique identifier for each record to be published (you can use the catalogue number, for example, or an existing barcode); * ''SourceInstitutionID'', an identifier for the institution publishing the dataset (for example BGBM); * ''SourceID'', an identifier for the dataset (e.g. “Herbarium Berolinense’’). The elements already shown in the mapping editor are printed in red because they are mandatory, which means they are required for an ABCD document. Without them being mapped (to database fields that are non-empty), the Provider Software will not be able to construct valid ABCD documents and will not publish any records. So the first elements that should be mapped are the mandatory fields. To add a mapping, click the <strong>+</strong> symbol next to the field, which will open the mapping editor dialog, where you can select the database table/column you want to map the ABCD element to: [[File:abcdMappingEditorDialog.png]] In this case, the element ''/DataSets/DataSet/Metadata/RevisionData/DateModified'' is mapped to the column ''source_update'' of the ''metadata'' table, which has the data type ''date''. (Remember that table/column retrieval is not supported for all DBMS, so if you’re unlucky, you might need to type in table/column name manually.) <small>Save</small> will accept your settings and close the dialog, <small>Cancel</small> will discard any changes. Instead of mapping an element to a database field, you can map it to a literal. For example, you could enter the technical and content contact information directly as literals in the mapping editor dialog (assuming they’re the same for all records). However, we strongly discourage you from doing this. For one thing, this is just bad style, since it will scatter the information published over different places: Some will be retrieved from the database, some (the literals) will be loaded from the configuration files. Changing the metadata of your service later would then require access to the database and access to the BioCASe configuration – that means the configuration tool password. Be nice to the person who will do these changes and put everything neatly in a database table! The second reason is even more important: Using literals for an element will result in this element to become not searchable, which means it cannot be used in a filter of a Search or a Scan request. Even though this behaviour is discoverable with a Capabilities request (by evaluating the ''searchable'' flag), a potential user of your web service might just not think of that, so it’s a good idea to avoid that in first place. Please use literals sparsely! In the mapping editor window, you can select several source columns and several literals, even a combination of both. The Provider Software will simply concatenate the values retrieved from the database and the literals specified (in the order they’re given). But again: Keep in mind that this will make the element not searchable. A better solution is to do any concatenation operations in a database view. Use the mapping editor window to add mappings for all eight mandatory fields: Click on the <strong>+</strong> symbol next to the element, choose table/column name and the correct data type, then press <small>Save</small>. When you’re done, the page should look similar to the following image. Press <small>Save</small> in the main window to write your changes to the configuration files. [[File:abcdMappingEditorMandatoryElements.png]] == Initial Testing of the Data Source == Once you’ve mapped all mandatory elements, you can do the first test of your BioCASe web service. Even though you haven’t mapped any real information yet, it is a good idea to check the service and solve any problems hitherto before continuing with the mapping. Click on <small>Test mapping!</small> to open the Query Form that can be used for this purpose: [[File:abcdQueryFormEmpty.png]] With the Query Form you can send a BioCASe request to your web service; it will wait for the reply of the web service and display the response document. On top of the page you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ''flora'' being the name of the datasource. Make sure this is correct for your installation! Into the text box below you can enter the request XML text. Because it would be cumbersome to enter the whole BioCASe request, you can use the links below the box to load templates. For now, click on <small>ABCD2 Search</small> to fill the box with a BioCASe Search request for documents in the ABCD2 format (the one we’ve just mapped): <pre> <?xml version='1.0' encoding='UTF-8'?> <request xmlns='http://www.biocase.org/schemas/protocol/1.3'> <header><type>search</type></header> <search> <requestFormat>http://www.tdwg.org/schemas/abcd/2.06</requestFormat> <responseFormat start='0' limit='10'>http://www.tdwg.org/schemas/abcd/2.06</responseFormat> <filter> <like path='/DataSets/DataSet/Units/Unit/Identifications/Identification/Result/ TaxonIdentified/ScientificName/FullScientificNameString'>A*</like> </filter> <count>false</count> </search> </request> </pre> As you can see, there is a filter set on the ''ScientificName'' element. Since we haven’t mapped this element yet, this won’t work. So let’s remove the filter (all lines between <filter> and </filter>), leaving the text box like this: [[File:abcdQueryFormFilled.png]] Pressing <small>Submit</small> will send the request to the web service and – hopefully – display the response document with the correct ABCD records. For the mapping we’ve just created, it should look similar to the following document (use the <strong>–</strong> symbol next to the trees TechnicalContact, ContentContact and Metadata to collapse them): [[File:abcdResponseMandatoryElements.png]] If you get an error, please check out the [[Debugging]] tutorial on how to debug a web service. You should continue the mapping process only after your web service returns results for the basal mapping we’ve just created. == Adding additional ABCD Elements == Once you’ve successfully managed to map the mandatory ABCD elements ("successfully" means the web service returns ABCD documents), you can go on with the mapping process. This is usually an iterative process: You add a handful of elements (for example, country, country code, locality text and coordinates for the gathering site), then you use the Query Form to see if the information ends up where it is supposed to do. This is the reason the Query Form opens up in a separate tab (or window, of you configured your web browser so): In one tab, you use the mapping editor to add new elements; after pressing <small>Save</small> you go to the Query Form tab and see what the response documents look like. Redo this until you’ve mapped all information you want to publish. In order to map new ABCD concepts, tick the check box labelled <small>Show all concepts</small> and press <small>Refresh</small>. The mapping editor will then display the whole ABCD tree, which looks admittedly a bit intimidating. In order to find the concept you’re looking for, use the search function of your browser. If you want to map the country name of the gathering site, for example, hit <small>Ctrl + F</small> (for Firefox, Microsoft IE and Opera) and type in the name of the ABCD element, namely ''country''. The browser will jump to the ''Country'' tree of the ''Gathering'' element, where you’ll find the elements for country name (''Name''), ISO country code (''ISO3166Code'') and the country name in the country’s language (''NameDerived''): [[File:abcdMappingEditorSearch.png|500px]] Continue the sequence mapping new elements/testing until you’ve mapped all information you want to publish. You can have a look at the list of recommend and commonly used ABCD elements on the [[CommonABCD2Concepts]] page. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki]. The following page will show you a sample response ABCD document for a web service with 77 concepts mapped. Of course your documents will look different, because you will probably use different ABCD elements, but still it’s good to get an idea. For easier viewing, you can save the contents of the box into a file with the extension .xml and open that in Firefix or MS Internet Explorer (don't use Safari). You will then be able to collapse the different XML sub trees: [[SampleABCDDocument]]. 64dc1c89e891ba2e2a919a69cad0f227d6de7437 6 53 147 2011-07-18T15:34:33Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 0 1 160 10 2011-07-21T11:06:28Z JoergHoletschek 3 wikitext text/x-wiki = Welcome to the Documentation Wiki of the BioCASe Provider Software = This is the Documentation Wiki of the BioCASe Provider Software. Here you will find tutorials on how to install, configure and debug the BioCASe Provider Software. More detailed information on BioCASe can be found in the [[BeginnersGuide|Beginner's Guide]] and on the [http://www.biocase.org/whats_biocase BioCase homepage]. [[File:BioCASeLogoLarge.jpg]] 67dbc98d557bb49e7a530c7a9e2270bd9f51c3fe 0 1 161 160 2011-07-21T11:06:48Z JoergHoletschek 3 wikitext text/x-wiki = Welcome to the Documentation Wiki of the BioCASe Provider Software = This is the Documentation Wiki of the BioCASe Provider Software. Here you will find tutorials on how to install, configure and debug the BioCASe Provider Software. More detailed information on BioCASe can be found in the [[BeginnersGuide|Beginner's Guide]] and on the [http://www.biocase.org/whats_biocase BioCASe homepage]. [[File:BioCASeLogoLarge.jpg]] b8b5cbf52805175e27249fd18a68d2fb7730bcde 163 161 2011-07-21T11:12:12Z JoergHoletschek 3 wikitext text/x-wiki = Welcome to the Documentation Wiki of the BioCASe Provider Software = [[Image:BioCASeLogoLarge.jpg|right|BioCASe]]This is the Documentation Wiki of the BioCASe Provider Software. Here you will find tutorials on how to install, configure and debug the BioCASe Provider Software. More detailed information on BioCASe can be found in the [[BeginnersGuide|Beginner's Guide]] and on the [http://www.biocase.org/whats_biocase BioCASe homepage]. 5cc463d51ccf1384a61fce6b1cf674793ed5c6e8 164 163 2011-07-21T11:18:33Z JoergHoletschek 3 wikitext text/x-wiki = Welcome to the Documentation Wiki of the BioCASe Provider Software = This is the Documentation Wiki of the BioCASe Provider Software. Here you will find tutorials on how to install, configure and debug the BioCASe Provider Software. More detailed information on BioCASe can be found in the [[BeginnersGuide|Beginner's Guide]] and on the [http://www.biocase.org/whats_biocase BioCASe homepage]. [[Image:BioCASeLogoLarge.jpg|200px|right|BioCASe]]Described in one paragraph, the BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. It is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. However, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the BioCASe network (http://www.biocase.org) and the Global Biodiversity Information Facility (http://www.gbif.org). After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. 0dfd20a2d7541bff6efd04c2b929d30c1740af95 165 164 2011-07-21T11:21:22Z JoergHoletschek 3 wikitext text/x-wiki = Welcome to the Documentation Wiki of the BioCASe Provider Software = This is the Documentation Wiki of the BioCASe Provider Software. Here you will find tutorials on how to install, configure and debug the BioCASe Provider Software. More detailed information on BioCASe can be found in the [[BeginnersGuide|Beginner's Guide]] and on the [http://www.biocase.org/whats_biocase BioCASe homepage]. [[Image:BioCASeLogoLarge.jpg|200px|right|BioCASe]]Described in one paragraph, the BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. It is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. However, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the BioCASe network (http://www.biocase.org) and the Global Biodiversity Information Facility (http://www.gbif.org). After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. == Table of Contents == [[BeginnersGuide|'''Beginner's Guide:''']] What the Provider Software is all about. [[BeginnersGuide|'''Beginner's Guide:''']] [[BeginnersGuide|'''Beginner's Guide:''']] [[BeginnersGuide|'''Beginner's Guide:''']] [[BeginnersGuide|'''Beginner's Guide:''']] [[BeginnersGuide|'''Beginner's Guide:''']] [[BeginnersGuide|'''Beginner's Guide:''']] [[BeginnersGuide|'''Beginner's Guide:''']] [[BeginnersGuide|'''Beginner's Guide:''']] [[BeginnersGuide|'''Beginner's Guide:''']] cc03e9636caaba3be193e8ddcc76602c7bd98a2b 166 165 2011-07-21T11:43:22Z JoergHoletschek 3 wikitext text/x-wiki = Welcome to the Documentation Wiki of the BioCASe Provider Software = This is the Documentation Wiki of the BioCASe Provider Software. Here you will find tutorials on how to install, configure and debug the BioCASe Provider Software. More detailed information on BioCASe can be found in the [[BeginnersGuide|Beginner's Guide]] and on the [http://www.biocase.org/whats_biocase BioCASe homepage]. [[Image:BioCASeLogoLarge.jpg|200px|right|BioCASe]]Described in one paragraph, the BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. It is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. However, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the BioCASe network (http://www.biocase.org) and the Global Biodiversity Information Facility (http://www.gbif.org). After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. == Feedback == If you have comments on the Provider Software, want to file a bug report or leave a suggestion for a useful new feature, please use the [[Comments|Comments page]] or [[Contact_the_BioCASe_Team|get in contact with the BioCASe Team]]. == Table of Contents == ;[[BeginnersGuide|Beginner's Guide]]: What the Provider Software is all about. ;[[Preparation]]: Things to be considered when publishing a database with BioCASe and potential preparatory steps needed. ;[[Installation]]: Tutorial on how to install the BioCASe Provider Software. ;[[DatasourceSetup|Datasource Setup]]: How to set up a new BioCASe web service (aka data source). ;[[ABCD2Mapping|ABCD2 Mapping]]: Tutorial on mapping ABCD2. ;[[CommonABCD2Concepts|CommonABCD2Concepts]]: List of mandatory, recommended and commonly used ABCD2 elements. ;[[SampleABCDDocument|Sample ABCD2 Document]]: Example ABCD2 document of a comprehensively mapped data provider. ;[[Debugging|Debugging Tutorial]]: Instructions on how to debug a BioCASe web service. ;[[Glossary|Glossary]]: List of terms and abbreviations used in this documentation. ;[[FAQ|FAQ]]: List of frequently asked questions and hopefully helpful answers. ;[[VersionHistory|Version History]]: Change list of current and past versions of the BioCASe Provider Software. 6be350315af0f68021c24c82b215b16e4a8d2676 167 166 2011-07-21T11:45:11Z JoergHoletschek 3 /* Feedback */ wikitext text/x-wiki = Welcome to the Documentation Wiki of the BioCASe Provider Software = This is the Documentation Wiki of the BioCASe Provider Software. Here you will find tutorials on how to install, configure and debug the BioCASe Provider Software. More detailed information on BioCASe can be found in the [[BeginnersGuide|Beginner's Guide]] and on the [http://www.biocase.org/whats_biocase BioCASe homepage]. [[Image:BioCASeLogoLarge.jpg|200px|right|BioCASe]]Described in one paragraph, the BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. It is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. However, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the BioCASe network (http://www.biocase.org) and the Global Biodiversity Information Facility (http://www.gbif.org). After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. == BioCASe Support == If you have comments on the Provider Software, want to file a bug report or leave a suggestion for a useful new feature, please use the [[Comments|Comments page]]. For getting help on BioCASe, please [[Contact_the_BioCASe_Team|get in contact with the BioCASe Team]]. == Table of Contents == ;[[BeginnersGuide|Beginner's Guide]]: What the Provider Software is all about. ;[[Preparation]]: Things to be considered when publishing a database with BioCASe and potential preparatory steps needed. ;[[Installation]]: Tutorial on how to install the BioCASe Provider Software. ;[[DatasourceSetup|Datasource Setup]]: How to set up a new BioCASe web service (aka data source). ;[[ABCD2Mapping|ABCD2 Mapping]]: Tutorial on mapping ABCD2. ;[[CommonABCD2Concepts|CommonABCD2Concepts]]: List of mandatory, recommended and commonly used ABCD2 elements. ;[[SampleABCDDocument|Sample ABCD2 Document]]: Example ABCD2 document of a comprehensively mapped data provider. ;[[Debugging|Debugging Tutorial]]: Instructions on how to debug a BioCASe web service. ;[[Glossary|Glossary]]: List of terms and abbreviations used in this documentation. ;[[FAQ|FAQ]]: List of frequently asked questions and hopefully helpful answers. ;[[VersionHistory|Version History]]: Change list of current and past versions of the BioCASe Provider Software. f9a396f1bf5a17b9931a38889f0a1aef9ff4d879 168 167 2011-07-21T11:46:04Z JoergHoletschek 3 /* Table of Contents */ wikitext text/x-wiki = Welcome to the Documentation Wiki of the BioCASe Provider Software = This is the Documentation Wiki of the BioCASe Provider Software. Here you will find tutorials on how to install, configure and debug the BioCASe Provider Software. More detailed information on BioCASe can be found in the [[BeginnersGuide|Beginner's Guide]] and on the [http://www.biocase.org/whats_biocase BioCASe homepage]. [[Image:BioCASeLogoLarge.jpg|200px|right|BioCASe]]Described in one paragraph, the BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. It is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. However, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the BioCASe network (http://www.biocase.org) and the Global Biodiversity Information Facility (http://www.gbif.org). After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. == BioCASe Support == If you have comments on the Provider Software, want to file a bug report or leave a suggestion for a useful new feature, please use the [[Comments|Comments page]]. For getting help on BioCASe, please [[Contact_the_BioCASe_Team|get in contact with the BioCASe Team]]. == Table of Contents == ;[[BeginnersGuide|Beginner's Guide]]: What the Provider Software is all about. ;[[Preparation]]: Things to be considered when publishing a database with BioCASe and potential preparatory steps needed. ;[[Installation]]: Tutorial on how to install the BioCASe Provider Software. ;[[DatasourceSetup|Datasource Setup]]: How to set up a new BioCASe web service (aka data source). ;[[ABCD2Mapping|ABCD2 Mapping]]: Tutorial on mapping ABCD2. ;[[CommonABCD2Concepts|Common ABCD2 Concepts]]: List of mandatory, recommended and commonly used ABCD2 elements. ;[[SampleABCDDocument|Sample ABCD2 Document]]: Example ABCD2 document of a comprehensively mapped data provider. ;[[Debugging|Debugging Tutorial]]: Instructions on how to debug a BioCASe web service. ;[[Glossary|Glossary]]: List of terms and abbreviations used in this documentation. ;[[FAQ|FAQ]]: List of frequently asked questions and hopefully helpful answers. ;[[VersionHistory|Version History]]: Change list of current and past versions of the BioCASe Provider Software. fea97aa4b1a52851b78fcdae700cc2ccb65cae0b 172 168 2011-07-21T13:43:12Z JoergHoletschek 3 wikitext text/x-wiki = Welcome to the Documentation Wiki of the BioCASe Provider Software = This is the Documentation Wiki of the BioCASe Provider Software. Here you will find tutorials on how to install, configure and debug the BioCASe Provider Software. More detailed information on BioCASe can be found in the [[BeginnersGuide|Beginner's Guide]] and on the [http://www.biocase.org/whats_biocase BioCASe homepage]. [[Image:BioCASeLogoLarge.jpg|200px|right|BioCASe]]Described in one paragraph, the BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. It is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. However, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the BioCASe network (http://www.biocase.org) and the Global Biodiversity Information Facility (http://www.gbif.org). After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. == BioCASe Support == [[Image:support_stamp.png|200px|right|BioCASe Support]]If you have comments on the Provider Software, want to file a bug report or leave a suggestion for a useful new feature, please use the [[Comments|Comments page]]. For getting help on BioCASe, please [[Contact_the_BioCASe_Team|get in contact with the BioCASe Team]]. == Table of Contents == ;[[BeginnersGuide|Beginner's Guide]]: What the Provider Software is all about. ;[[Preparation]]: Things to be considered when publishing a database with BioCASe and potential preparatory steps needed. ;[[Installation]]: Tutorial on how to install the BioCASe Provider Software. ;[[DatasourceSetup|Datasource Setup]]: How to set up a new BioCASe web service (aka data source). ;[[ABCD2Mapping|ABCD2 Mapping]]: Tutorial on mapping ABCD2. ;[[CommonABCD2Concepts|Common ABCD2 Concepts]]: List of mandatory, recommended and commonly used ABCD2 elements. ;[[SampleABCDDocument|Sample ABCD2 Document]]: Example ABCD2 document of a comprehensively mapped data provider. ;[[Debugging|Debugging Tutorial]]: Instructions on how to debug a BioCASe web service. ;[[Glossary|Glossary]]: List of terms and abbreviations used in this documentation. ;[[FAQ|FAQ]]: List of frequently asked questions and hopefully helpful answers. ;[[VersionHistory|Version History]]: Change list of current and past versions of the BioCASe Provider Software. fda4a950b501ac5ebbf331226f3ae474a67d9c47 174 172 2011-07-21T13:51:59Z JoergHoletschek 3 wikitext text/x-wiki = Welcome to the Documentation Wiki of the BioCASe Provider Software = This is the Documentation Wiki of the BioCASe Provider Software. Here you will find tutorials on how to install, configure and debug the BioCASe Provider Software. More detailed information on BioCASe can be found in the [[BeginnersGuide|Beginner's Guide]] and on the [http://www.biocase.org/whats_biocase BioCASe homepage]. [[Image:BioCASeLogoLarge.jpg|link=http://www.biocase.org|200px|right|BioCASe]]Described in one paragraph, the BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. It is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. However, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the BioCASe network (http://www.biocase.org) and the Global Biodiversity Information Facility (http://www.gbif.org). After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. == BioCASe Support == If you have comments on the Provider Software, want to file a bug report or leave a suggestion for a useful new feature, please use the [[Comments|Comments page]]. For getting help on BioCASe, please [[Contact_the_BioCASe_Team|get in contact with the BioCASe Team]]. == Table of Contents == ;[[BeginnersGuide|Beginner's Guide]]: What the Provider Software is all about. ;[[Preparation]]: Things to be considered when publishing a database with BioCASe and potential preparatory steps needed. [[Image:support_stamp.png|link=Contact_the_BioCASe_Team|200px|right|BioCASe Support]] ;[[Installation]]: Tutorial on how to install the BioCASe Provider Software. ;[[DatasourceSetup|Datasource Setup]]: How to set up a new BioCASe web service (aka data source). ;[[ABCD2Mapping|ABCD2 Mapping]]: Tutorial on mapping ABCD2. ;[[CommonABCD2Concepts|Common ABCD2 Concepts]]: List of mandatory, recommended and commonly used ABCD2 elements. ;[[SampleABCDDocument|Sample ABCD2 Document]]: Example ABCD2 document of a comprehensively mapped data provider. ;[[Debugging|Debugging Tutorial]]: Instructions on how to debug a BioCASe web service. ;[[Glossary|Glossary]]: List of terms and abbreviations used in this documentation. ;[[FAQ|FAQ]]: List of frequently asked questions and hopefully helpful answers. ;[[VersionHistory|Version History]]: Change list of current and past versions of the BioCASe Provider Software. 67b81ded59cc5d046502e6d9e975dd1333cece55 175 174 2011-07-21T13:53:59Z JoergHoletschek 3 /* BioCASe Support */ wikitext text/x-wiki = Welcome to the Documentation Wiki of the BioCASe Provider Software = This is the Documentation Wiki of the BioCASe Provider Software. Here you will find tutorials on how to install, configure and debug the BioCASe Provider Software. More detailed information on BioCASe can be found in the [[BeginnersGuide|Beginner's Guide]] and on the [http://www.biocase.org/whats_biocase BioCASe homepage]. [[Image:BioCASeLogoLarge.jpg|link=http://www.biocase.org|200px|right|BioCASe]]Described in one paragraph, the BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. It is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. However, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the BioCASe network (http://www.biocase.org) and the Global Biodiversity Information Facility (http://www.gbif.org). After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. == BioCASe Support == If you have questions, comments, want to file a bug report or leave a suggestion for a useful new feature, you can * use the [[Comments|Comments page]] to leave a comment, or * [[Contact_the_BioCASe_Team|get in contact with the BioCASe Team]] directly. == Table of Contents == ;[[BeginnersGuide|Beginner's Guide]]: What the Provider Software is all about. ;[[Preparation]]: Things to be considered when publishing a database with BioCASe and potential preparatory steps needed. [[Image:support_stamp.png|link=Contact_the_BioCASe_Team|200px|right|BioCASe Support]] ;[[Installation]]: Tutorial on how to install the BioCASe Provider Software. ;[[DatasourceSetup|Datasource Setup]]: How to set up a new BioCASe web service (aka data source). ;[[ABCD2Mapping|ABCD2 Mapping]]: Tutorial on mapping ABCD2. ;[[CommonABCD2Concepts|Common ABCD2 Concepts]]: List of mandatory, recommended and commonly used ABCD2 elements. ;[[SampleABCDDocument|Sample ABCD2 Document]]: Example ABCD2 document of a comprehensively mapped data provider. ;[[Debugging|Debugging Tutorial]]: Instructions on how to debug a BioCASe web service. ;[[Glossary|Glossary]]: List of terms and abbreviations used in this documentation. ;[[FAQ|FAQ]]: List of frequently asked questions and hopefully helpful answers. ;[[VersionHistory|Version History]]: Change list of current and past versions of the BioCASe Provider Software. 770fe4cc8c9f11533cbeef3f6e18bb02441c4b29 176 175 2011-07-21T13:55:20Z JoergHoletschek 3 /* BioCASe Support */ wikitext text/x-wiki = Welcome to the Documentation Wiki of the BioCASe Provider Software = This is the Documentation Wiki of the BioCASe Provider Software. Here you will find tutorials on how to install, configure and debug the BioCASe Provider Software. More detailed information on BioCASe can be found in the [[BeginnersGuide|Beginner's Guide]] and on the [http://www.biocase.org/whats_biocase BioCASe homepage]. [[Image:BioCASeLogoLarge.jpg|link=http://www.biocase.org|200px|right|BioCASe]]Described in one paragraph, the BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. It is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. However, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the BioCASe network (http://www.biocase.org) and the Global Biodiversity Information Facility (http://www.gbif.org). After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. == BioCASe Support == [[Image:mail.png|link=Contact_the_BioCASe_Team|200px|right|BioCASe Support]] If you have questions, comments, want to file a bug report or leave a suggestion for a useful new feature, you can * use the [[Comments|Comments page]] to leave a comment, or * [[Contact_the_BioCASe_Team|get in contact with the BioCASe Team]] directly. == Table of Contents == ;[[BeginnersGuide|Beginner's Guide]]: What the Provider Software is all about. ;[[Preparation]]: Things to be considered when publishing a database with BioCASe and potential preparatory steps needed. [[Image:support_stamp.png|link=Contact_the_BioCASe_Team|200px|right|BioCASe Support]] ;[[Installation]]: Tutorial on how to install the BioCASe Provider Software. ;[[DatasourceSetup|Datasource Setup]]: How to set up a new BioCASe web service (aka data source). ;[[ABCD2Mapping|ABCD2 Mapping]]: Tutorial on mapping ABCD2. ;[[CommonABCD2Concepts|Common ABCD2 Concepts]]: List of mandatory, recommended and commonly used ABCD2 elements. ;[[SampleABCDDocument|Sample ABCD2 Document]]: Example ABCD2 document of a comprehensively mapped data provider. ;[[Debugging|Debugging Tutorial]]: Instructions on how to debug a BioCASe web service. ;[[Glossary|Glossary]]: List of terms and abbreviations used in this documentation. ;[[FAQ|FAQ]]: List of frequently asked questions and hopefully helpful answers. ;[[VersionHistory|Version History]]: Change list of current and past versions of the BioCASe Provider Software. f21c96b658d053a132acf0632c3b5a8cf5f7abc8 178 176 2011-07-21T13:56:17Z JoergHoletschek 3 /* BioCASe Support */ wikitext text/x-wiki = Welcome to the Documentation Wiki of the BioCASe Provider Software = This is the Documentation Wiki of the BioCASe Provider Software. Here you will find tutorials on how to install, configure and debug the BioCASe Provider Software. More detailed information on BioCASe can be found in the [[BeginnersGuide|Beginner's Guide]] and on the [http://www.biocase.org/whats_biocase BioCASe homepage]. [[Image:BioCASeLogoLarge.jpg|link=http://www.biocase.org|200px|right|BioCASe]]Described in one paragraph, the BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. It is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. However, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the BioCASe network (http://www.biocase.org) and the Global Biodiversity Information Facility (http://www.gbif.org). After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. == BioCASe Support == [[Image:mail.png|link=Contact_the_BioCASe_Team|150px|right|BioCASe Support]] If you have questions, comments, want to file a bug report or leave a suggestion for a useful new feature, you can * use the [[Comments|Comments page]] to leave a comment, or * [[Contact_the_BioCASe_Team|get in contact with the BioCASe Team]] directly. == Table of Contents == ;[[BeginnersGuide|Beginner's Guide]]: What the Provider Software is all about. ;[[Preparation]]: Things to be considered when publishing a database with BioCASe and potential preparatory steps needed. [[Image:support_stamp.png|link=Contact_the_BioCASe_Team|200px|right|BioCASe Support]] ;[[Installation]]: Tutorial on how to install the BioCASe Provider Software. ;[[DatasourceSetup|Datasource Setup]]: How to set up a new BioCASe web service (aka data source). ;[[ABCD2Mapping|ABCD2 Mapping]]: Tutorial on mapping ABCD2. ;[[CommonABCD2Concepts|Common ABCD2 Concepts]]: List of mandatory, recommended and commonly used ABCD2 elements. ;[[SampleABCDDocument|Sample ABCD2 Document]]: Example ABCD2 document of a comprehensively mapped data provider. ;[[Debugging|Debugging Tutorial]]: Instructions on how to debug a BioCASe web service. ;[[Glossary|Glossary]]: List of terms and abbreviations used in this documentation. ;[[FAQ|FAQ]]: List of frequently asked questions and hopefully helpful answers. ;[[VersionHistory|Version History]]: Change list of current and past versions of the BioCASe Provider Software. 7febb2ab0c8c9037dfa07968ffd6322aa05f8353 179 178 2011-07-21T14:00:33Z JoergHoletschek 3 /* BioCASe Support */ wikitext text/x-wiki = Welcome to the Documentation Wiki of the BioCASe Provider Software = This is the Documentation Wiki of the BioCASe Provider Software. Here you will find tutorials on how to install, configure and debug the BioCASe Provider Software. More detailed information on BioCASe can be found in the [[BeginnersGuide|Beginner's Guide]] and on the [http://www.biocase.org/whats_biocase BioCASe homepage]. [[Image:BioCASeLogoLarge.jpg|link=http://www.biocase.org|200px|right|BioCASe]]Described in one paragraph, the BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. It is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. However, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the BioCASe network (http://www.biocase.org) and the Global Biodiversity Information Facility (http://www.gbif.org). After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. == BioCASe Support == [[Image:mail.png|link=Contact_the_BioCASe_Team|150px|right|BioCASe Support]] You can use the [[Comments|Comments page]] to just leave us a comment. If you want us to reply to your feedback, you should [[Contact_the_BioCASe_Team|get in contact with the BioCASe Team]] directly. Please do so if you * have any questions concerning BioCASe in general or the installation/configuration of the Provider Software, * want to file a bug report, * have any comments, or * want to make a suggestion for a useful new feature in upcoming versions. == Table of Contents == ;[[BeginnersGuide|Beginner's Guide]]: What the Provider Software is all about. ;[[Preparation]]: Things to be considered when publishing a database with BioCASe and potential preparatory steps needed. [[Image:support_stamp.png|link=Contact_the_BioCASe_Team|200px|right|BioCASe Support]] ;[[Installation]]: Tutorial on how to install the BioCASe Provider Software. ;[[DatasourceSetup|Datasource Setup]]: How to set up a new BioCASe web service (aka data source). ;[[ABCD2Mapping|ABCD2 Mapping]]: Tutorial on mapping ABCD2. ;[[CommonABCD2Concepts|Common ABCD2 Concepts]]: List of mandatory, recommended and commonly used ABCD2 elements. ;[[SampleABCDDocument|Sample ABCD2 Document]]: Example ABCD2 document of a comprehensively mapped data provider. ;[[Debugging|Debugging Tutorial]]: Instructions on how to debug a BioCASe web service. ;[[Glossary|Glossary]]: List of terms and abbreviations used in this documentation. ;[[FAQ|FAQ]]: List of frequently asked questions and hopefully helpful answers. ;[[VersionHistory|Version History]]: Change list of current and past versions of the BioCASe Provider Software. 73dca82abe2e722964ab2417dd89d06cb0c749c8 180 179 2011-07-21T14:00:44Z JoergHoletschek 3 /* BioCASe Support */ wikitext text/x-wiki = Welcome to the Documentation Wiki of the BioCASe Provider Software = This is the Documentation Wiki of the BioCASe Provider Software. Here you will find tutorials on how to install, configure and debug the BioCASe Provider Software. More detailed information on BioCASe can be found in the [[BeginnersGuide|Beginner's Guide]] and on the [http://www.biocase.org/whats_biocase BioCASe homepage]. [[Image:BioCASeLogoLarge.jpg|link=http://www.biocase.org|200px|right|BioCASe]]Described in one paragraph, the BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. It is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. However, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the BioCASe network (http://www.biocase.org) and the Global Biodiversity Information Facility (http://www.gbif.org). After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. == BioCASe Support == [[Image:mail.png|link=Contact_the_BioCASe_Team|150px|left|BioCASe Support]] You can use the [[Comments|Comments page]] to just leave us a comment. If you want us to reply to your feedback, you should [[Contact_the_BioCASe_Team|get in contact with the BioCASe Team]] directly. Please do so if you * have any questions concerning BioCASe in general or the installation/configuration of the Provider Software, * want to file a bug report, * have any comments, or * want to make a suggestion for a useful new feature in upcoming versions. == Table of Contents == ;[[BeginnersGuide|Beginner's Guide]]: What the Provider Software is all about. ;[[Preparation]]: Things to be considered when publishing a database with BioCASe and potential preparatory steps needed. [[Image:support_stamp.png|link=Contact_the_BioCASe_Team|200px|right|BioCASe Support]] ;[[Installation]]: Tutorial on how to install the BioCASe Provider Software. ;[[DatasourceSetup|Datasource Setup]]: How to set up a new BioCASe web service (aka data source). ;[[ABCD2Mapping|ABCD2 Mapping]]: Tutorial on mapping ABCD2. ;[[CommonABCD2Concepts|Common ABCD2 Concepts]]: List of mandatory, recommended and commonly used ABCD2 elements. ;[[SampleABCDDocument|Sample ABCD2 Document]]: Example ABCD2 document of a comprehensively mapped data provider. ;[[Debugging|Debugging Tutorial]]: Instructions on how to debug a BioCASe web service. ;[[Glossary|Glossary]]: List of terms and abbreviations used in this documentation. ;[[FAQ|FAQ]]: List of frequently asked questions and hopefully helpful answers. ;[[VersionHistory|Version History]]: Change list of current and past versions of the BioCASe Provider Software. 825e45bc9cc650c72fcd8253a2ef229620847ec6 181 180 2011-07-21T14:01:17Z JoergHoletschek 3 /* BioCASe Support */ wikitext text/x-wiki = Welcome to the Documentation Wiki of the BioCASe Provider Software = This is the Documentation Wiki of the BioCASe Provider Software. Here you will find tutorials on how to install, configure and debug the BioCASe Provider Software. More detailed information on BioCASe can be found in the [[BeginnersGuide|Beginner's Guide]] and on the [http://www.biocase.org/whats_biocase BioCASe homepage]. [[Image:BioCASeLogoLarge.jpg|link=http://www.biocase.org|200px|right|BioCASe]]Described in one paragraph, the BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. It is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. However, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the BioCASe network (http://www.biocase.org) and the Global Biodiversity Information Facility (http://www.gbif.org). After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. == BioCASe Support == [[Image:mail.png|link=Contact_the_BioCASe_Team|150px|right|BioCASe Support]] You can use the [[Comments|Comments page]] to just leave us a comment. If you want us to reply to your feedback, you should [[Contact_the_BioCASe_Team|get in contact with the BioCASe Team]] directly. Please do so if you * have any questions concerning BioCASe in general or the installation/configuration of the Provider Software, * want to file a bug report, * have any comments, or * want to make a suggestion for a useful new feature in upcoming versions. == Table of Contents == ;[[BeginnersGuide|Beginner's Guide]]: What the Provider Software is all about. ;[[Preparation]]: Things to be considered when publishing a database with BioCASe and potential preparatory steps needed. [[Image:support_stamp.png|link=Contact_the_BioCASe_Team|200px|right|BioCASe Support]] ;[[Installation]]: Tutorial on how to install the BioCASe Provider Software. ;[[DatasourceSetup|Datasource Setup]]: How to set up a new BioCASe web service (aka data source). ;[[ABCD2Mapping|ABCD2 Mapping]]: Tutorial on mapping ABCD2. ;[[CommonABCD2Concepts|Common ABCD2 Concepts]]: List of mandatory, recommended and commonly used ABCD2 elements. ;[[SampleABCDDocument|Sample ABCD2 Document]]: Example ABCD2 document of a comprehensively mapped data provider. ;[[Debugging|Debugging Tutorial]]: Instructions on how to debug a BioCASe web service. ;[[Glossary|Glossary]]: List of terms and abbreviations used in this documentation. ;[[FAQ|FAQ]]: List of frequently asked questions and hopefully helpful answers. ;[[VersionHistory|Version History]]: Change list of current and past versions of the BioCASe Provider Software. 73dca82abe2e722964ab2417dd89d06cb0c749c8 182 181 2011-07-21T14:01:38Z JoergHoletschek 3 /* Welcome to the Documentation Wiki of the BioCASe Provider Software */ wikitext text/x-wiki = Welcome to the Documentation Wiki of the BioCASe Provider Software = This is the Documentation Wiki of the BioCASe Provider Software. Here you will find tutorials on how to install, configure and debug the BioCASe Provider Software. More detailed information on BioCASe can be found in the [[BeginnersGuide|Beginner's Guide]] and on the [http://www.biocase.org/whats_biocase BioCASe homepage]. [[Image:BioCASeLogoLarge.jpg|link=http://www.biocase.org|200px|left|BioCASe]]Described in one paragraph, the BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. It is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. However, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the BioCASe network (http://www.biocase.org) and the Global Biodiversity Information Facility (http://www.gbif.org). After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. == BioCASe Support == [[Image:mail.png|link=Contact_the_BioCASe_Team|150px|right|BioCASe Support]] You can use the [[Comments|Comments page]] to just leave us a comment. If you want us to reply to your feedback, you should [[Contact_the_BioCASe_Team|get in contact with the BioCASe Team]] directly. Please do so if you * have any questions concerning BioCASe in general or the installation/configuration of the Provider Software, * want to file a bug report, * have any comments, or * want to make a suggestion for a useful new feature in upcoming versions. == Table of Contents == ;[[BeginnersGuide|Beginner's Guide]]: What the Provider Software is all about. ;[[Preparation]]: Things to be considered when publishing a database with BioCASe and potential preparatory steps needed. [[Image:support_stamp.png|link=Contact_the_BioCASe_Team|200px|right|BioCASe Support]] ;[[Installation]]: Tutorial on how to install the BioCASe Provider Software. ;[[DatasourceSetup|Datasource Setup]]: How to set up a new BioCASe web service (aka data source). ;[[ABCD2Mapping|ABCD2 Mapping]]: Tutorial on mapping ABCD2. ;[[CommonABCD2Concepts|Common ABCD2 Concepts]]: List of mandatory, recommended and commonly used ABCD2 elements. ;[[SampleABCDDocument|Sample ABCD2 Document]]: Example ABCD2 document of a comprehensively mapped data provider. ;[[Debugging|Debugging Tutorial]]: Instructions on how to debug a BioCASe web service. ;[[Glossary|Glossary]]: List of terms and abbreviations used in this documentation. ;[[FAQ|FAQ]]: List of frequently asked questions and hopefully helpful answers. ;[[VersionHistory|Version History]]: Change list of current and past versions of the BioCASe Provider Software. bcfea20ca719c0e65ff7ddb18f87d4990344b5f6 183 182 2011-07-21T14:08:23Z JoergHoletschek 3 wikitext text/x-wiki = Welcome to the Documentation Wiki of the BioCASe Provider Software = This is the Documentation Wiki of the BioCASe Provider Software. Here you will find tutorials on how to install, configure and debug the BioCASe Provider Software. More detailed information on BioCASe can be found in the [[BeginnersGuide|Beginner's Guide]] and on the [http://www.biocase.org/whats_biocase BioCASe homepage]. [[Image:BioCASeLogoLarge.jpg|link=http://www.biocase.org|200px|left|BioCASe]]Described in one paragraph, the BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. It is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. However, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the BioCASe network (http://www.biocase.org) and the Global Biodiversity Information Facility (http://www.gbif.org). After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. == BioCASe Support == [[Image:mail.png|link=Contact_the_BioCASe_team|150px|right|BioCASe Support]] You can use the [[Comments|Comments page]] to just leave us a comment. If you want us to reply to your feedback, you should [[Contact_the_BioCASe_team|get in contact with the BioCASe Team]] directly. Please do so if you * have any questions concerning BioCASe in general or the installation/configuration of the Provider Software, * want to file a bug report, * have any comments, or * want to make a suggestion for a useful new feature in upcoming versions. == Table of Contents == ;[[BeginnersGuide|Beginner's Guide]]: What the Provider Software is all about. ;[[Preparation]]: Things to be considered when publishing a database with BioCASe and potential preparatory steps needed. [[Image:support_stamp.png|link=Contact_the_BioCASe_team|200px|right|BioCASe Support]] ;[[Installation]]: Tutorial on how to install the BioCASe Provider Software. ;[[DatasourceSetup|Datasource Setup]]: How to set up a new BioCASe web service (aka data source). ;[[ABCD2Mapping|ABCD2 Mapping]]: Tutorial on mapping ABCD2. ;[[CommonABCD2Concepts|Common ABCD2 Concepts]]: List of mandatory, recommended and commonly used ABCD2 elements. ;[[SampleABCDDocument|Sample ABCD2 Document]]: Example ABCD2 document of a comprehensively mapped data provider. ;[[Debugging|Debugging Tutorial]]: Instructions on how to debug a BioCASe web service. ;[[Glossary|Glossary]]: List of terms and abbreviations used in this documentation. ;[[FAQ|FAQ]]: List of frequently asked questions and hopefully helpful answers. ;[[VersionHistory|Version History]]: Change list of current and past versions of the BioCASe Provider Software. 6218e96bac565eb48164026427e236e0d736993f 200 183 2011-07-21T15:23:18Z JoergHoletschek 3 wikitext text/x-wiki = Welcome to the Documentation Wiki of the BioCASe Provider Software = This is the Documentation Wiki of the BioCASe Provider Software. Here you will find tutorials on how to install, configure and debug the BioCASe Provider Software. More detailed information on BioCASe can be found in the [[BeginnersGuide|Beginner's Guide]] and on the [http://www.biocase.org/whats_biocase BioCASe homepage]. [[Image:BioCASeLogoLarge.jpg|link=http://www.biocase.org|150px|right|BioCASe]]Described in one paragraph, the BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. It is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. However, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the BioCASe network (http://www.biocase.org) and the Global Biodiversity Information Facility (http://www.gbif.org). After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. == BioCASe Support == [[Image:mail.png|link=Contact_the_BioCASe_team|150px|right|BioCASe Support]] You can use the [[Comments|Comments page]] to just leave us a comment. If you want us to reply to your feedback, you should [[Contact_the_BioCASe_team|get in contact with the BioCASe Team]] directly. Please do so if you * have any questions concerning BioCASe in general or the installation/configuration of the Provider Software, * want to file a bug report, * have any comments, or * want to make a suggestion for a useful new feature in upcoming versions. == Table of Contents == ;[[BeginnersGuide|Beginner's Guide]]: What the Provider Software is all about. ;[[Preparation]]: Things to be considered when publishing a database with BioCASe and potential preparatory steps needed. [[Image:support_stamp.png|link=Contact_the_BioCASe_team|200px|right|BioCASe Support]] ;[[Installation]]: Tutorial on how to install the BioCASe Provider Software. ;[[DatasourceSetup|Datasource Setup]]: How to set up a new BioCASe web service (aka data source). ;[[ABCD2Mapping|ABCD2 Mapping]]: Tutorial on mapping ABCD2. ;[[CommonABCD2Concepts|Common ABCD2 Concepts]]: List of mandatory, recommended and commonly used ABCD2 elements. ;[[SampleABCDDocument|Sample ABCD2 Document]]: Example ABCD2 document of a comprehensively mapped data provider. ;[[Debugging|Debugging Tutorial]]: Instructions on how to debug a BioCASe web service. ;[[Glossary|Glossary]]: List of terms and abbreviations used in this documentation. ;[[FAQ|FAQ]]: List of frequently asked questions and hopefully helpful answers. ;[[VersionHistory|Version History]]: Change list of current and past versions of the BioCASe Provider Software. f1e27cf1acbfabdc897ba6bb348c177ba9b249fe 6 55 162 2011-07-21T11:07:17Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 0 24 169 144 2011-07-21T12:49:36Z JoergHoletschek 3 /* Setting up the database connection */ wikitext text/x-wiki This tutorial explains in detail how to set up a datasource. == A BioCASe Data Source == Publishing a database with BioCASe means it will be made accessible through a BioCASe web service, which will be used later by the biodiversity network(s) you want to join to access to your data. For example, the GBIF Harvester will send a BioCASe Scan request to that web service to get a list of all scientific names kept in the database, and subsequently several BioCASe Search requests to load all records for these names. For each database you want to publish, you need to set up a BioCASe data source (do not mix that up with ODBC data sources on Windows machines). The resulting BioCASe web service is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source. So if you made your installation available at http://www.foobar.org/biocase during the installation process and set up a data source named Herbar, the URL of the BioCASe web service would be http://www.foobar.org/biocase/pywrapper.cgi?dsa=Herbar (data source names can be case sensitive, depending on your server’s operating system). == Creating a New Data Source == The start page of the BPS will always list the data sources currently configured, that means the web services offered by that BioCASe installation. On a vanilla installation, it will be empty: [[File:instBiocaseStartpage.png|500px]] Creating a new data source can be done in the configuration tool. Choose <small>Config Tool</small> on the start page, then <small>System Administration</small>. Log in with the password you set during the installation process (if you didn’t change that yet, the default is ''ACDC''). In the configuration screen, the sections <small>Data Sources</small> and <small>Recycle Bin</small> can be used to create, remove und un-remove a data source: [[File:dsaCreateDatasourceEmpty.png|500px]] For now, we want to create a new, empty data source. Enter a name – without any spaces – in the text box and click <small>Create DSA</small> (remember that the datasource name is case sensitive on Linux machines). After a second, it will appear in the list of existing data sources with a clickable link. By doing this, you created the first BioCASe web service for your installation. Clicking <small>Remove</small> will uninstall this service again (putting it in the recycle bin, so you can restore it later). Tip: If you select <small>abcdmetadata</small> in the template drop-down list, your data source will be created with an ABCD mapping for all the metadata elements stored in the metadata table mentioned in the [[Preparation]] guide. So if you did use the templates in ''sampledb/abcdmetadata'' to create the metadata table in your database, you can use this template to spare the hassle of mapping this table and start right away with your occurrence data. == Setting up the database connection == Clicking on the data source you’ve just created will take you to the datasource configuration tool. Note that the <small>Schemas</small> section will be empty if you’ve created an empty data source or list two schemas (ABCD 1.2 and 2.06) if you created the datasource based on the <small>abcdmetadata</small> template: [[File:dsaOverview.png|450px]] The link in the <small>Database Connection</small> section will take you to the connection parameters dialog. Fill in the values for your database, press <small>Save</small> and hope for the connection to turn into a nice green <small>OK</small>. '''DBMS:''' Select the entry appropriate for your collection. The BPS can directly connect to database servers (MySQL, Postgres, DB2, SQL Server, and Oracle) and to Desktop-DBMS like Access and FoxPro through ODBC. ODBC is only available on Windows, so if you plan to use it, BioCASe must be installed on a Windows Server. '''Host:''' Enter the IP address of the database server. If you’re connection through ODBC, you must leave this empty. Instead, use Windows’ ODBC Administrator to set up a '''System datasource''' (!); the server or file name of the database will be stored in the ODBC configuration. '''Database:''' Enter the name of the ODBC datasource if you’re using ODBC, the name of the database if you’re directly connecting to a server. '''User/Password:''' Database credentials to be used. As written in the [[Preparation]] guide, you should create a user with just read access for this. '''Encoding:''' The encoding of your database. If you’re not sure, leave the default, you can change it if you run into problems later. If you know your database is using a different encoding than latin-1 (which is typically the default for western languages), maybe UTF-8, select the appropriate entry in the drop-down list. [[File:dsaDBConnection.png|450px]] Once you’ve entered the correct values, press <small>Save</small>. Click <small>Overview</small> in the link list at the top of the page to return to the datasource configuration page. If the connection status doesn’t turn into <small>OK</small> (and doesn’t after you’ve returned to the overview page), please read the [[Debugging]] guide about how to trace the problem. == Setting up the Database Structure == Clicking on <small>Edit DB Structure</small> on the datasource overview page will show the DB structure setup. Make sure the connection status at the top of the page still is <small>OK</small>. In a first step, you will add all tables/views that hold information you want to publish with BioCASe. You can ignore all database tables storing only data you don’t plan to map to ABCD. Same holds true if you use cache tables (as explained in the [[Preparation]] guide): Only the cache tables need to be set up on the database structure page. In a second step, you will specify the foreign key references between these tables. === Adding Tables/Views with their Primary Keys === To add a table or view, just select it in the drop-down list. The name of the table will be filled in the <small>Alias</small> text box (you will use this alias later in the mapping process), feel free to change it into something that describes the contents of that tables (for example ''specimen'' for the root specimen table). Press <small>Add</small> to insert the table into the list. Before continuing to the next table, select the primary key in the list of attributes under <small>Primary Key Attribute(s)</small>. In case the data type is a string and not an integer, select <small>text</small> from the drop-down list. Press <small>Add</small> to add the primary key. If the primary key consists of several columns, you can add several primary key attributes. Redo this step (adding the tables first, then the primary key(s)) for all tables you want to publish data from. Always remember to set the correct primary key column(s), even if they’re not defined as primary keys in the database. Tables without a primary key cannot be used by the Provider Software and will be removed upon saving! === Specifying Foreign Key References === Once you’ve added all tables, you can define the relations between these tables by defining foreign keys. In order to do this, go to the <small>Foreign Key(s)</small> column of the table that holds a foreign key reference to another table, select the alias of the referenced table and press <small>Add</small>. Select the attribute that stores the foreign key and change the data type if it is not integer, then press <small>Add</small>. You must specify as many attributes for the foreign keys as the primary key of the referenced table has, otherwise the Provider Software won’t be able to create a valid join between the tables. Also, they must be in the same order and have identical data types. Redo this for all tables until you’ve specified all foreign key references. If you’ve installed the Graphviz package, the Provider Software will create a graph visualizing the current database structure. Each table will be visible as a node with the primary key(s) listed. References between tables will be represented by arrows, the foreign key(s) listed in the referring table: [[File:dsaDatabaseStructure_AlgenEngels.png|800px]] === Saving, Editing an Existing Setup === You can change aliases of tables you’ve already added to the list. To do so, go to the <small>Alias</small> text box of that table and type in a different name. Upon exiting the text box, all references to this alias will be updated in the table setup; also the graph will reflect the changes. But caution: Schema mappings use these aliases to bind schema elements to tables and columns, if you change an alias you’ve already used in a schema mapping, you will break this binding! So it is advisable to decide for meaningful alias names and keep them stable once you’ve started the mapping process. Once you’ve added a table/view to the table list, you can’t change the underlying table, since this would invalidate all the primary and foreign key attributes. If you’ve added the wrong table, remove it by using the trash can symbol next to <small>Delete Alias</small> and add the correct table/view. If you’ve finished the table setup and the graph displays the model of your database correctly, save the configuration by pressing <small>Save</small> at the bottom of the page. This is important; navigating away from the page without saving will cause your changes to be lost (even though you will be asked for confirmation)! It is advisable to save the table setup every now and then, for example once you’ve added all tables and primary keys. If you edit an existing table setup and happen to realize you’ve messed it up, you can revert to the state when you’ve pressed Save for the last time by using the <small>Revert</small> button at the bottom. But be careful: All unsaved changes will be lost. '''<u>Important Note:</u>''' The retrieval of tables/view/column names is currently only supported for the following DBMS: Microsoft SQL Server, MySQL, Access, Postgres and FoxPro. If you use a different DBMS, you will see a simple text box instead of a drop-down-list, where you’ll have to type in the name of the table or column you want to add. 7e4a2f5472f0c035d9b97a73cb6e4d214d5b8d97 0 17 170 137 2011-07-21T13:28:26Z JoergHoletschek 3 /* Metadata */ wikitext text/x-wiki Before you start publishing your database with BioCASe, you need to consider several issues. In some cases, some preparation of your database for the publication might be necessary. You should read this page of the wiki careful before you start configuring the Provider Software for a given database. == Live-DB versus Snapshot == The easiest way is to connect BioCASe to the same database you’re working with. This would result in any changes you do being visible on the BioCASe access point right away – which is an idea you might not like. Some curators prefer to publish defined versions of their database, for example after they’ve finished reviewing all specimens of a certain genus. If you prefer this, you need to create a second database that holds defined snapshots of your working database and that will be published instead. Snapshots could be generated in a regular interval (e.g., each Friday evening) or upon request. Please contact your IT guys for more help on this. == Access Control == The BioCASe Provider Software will only publish your data to biodiversity networks, meaning it will only read from the database and never do any changes. For security reasons it’s a good idea to create a user for BioCASe that has only read privileges on the database – more precisely on the tables you want to publish. Since the configuration tool will try to load a list of tables and columns from the database to allow easier setup, this user should also be allowed to load meta-information (<small>SHOW TABLES, SHOW COLUMNS</small>). == Metadata == Metadata describe your dataset as a whole - title, description, curator’s name, ownership, date of last modification, terms of use etc. BioCASe can publish a lot of metadata – and you should take the time to provide as much information as you can, since this will increase the usability of your published data. For storing metadata, please create an extra table in your database and fill it in. If you go to the BioCASe folder of your installation, you’ll find the template for this table in ''sampledb/abcdmetadata''. Either use the SQL script to create it in your database, or import it from the Access database (''metadata.mdb''). Per default, all records of your database will make up one dataset, being described by one set of metadata. In case your database holds records of several datasets, you can also group these into several datasets in ABCD by providing different sets of metadata. So when you realize during gathering of meta-information that they’re not the same for all records, this is an indicator that your database must be grouped into several datasets. If all records to be published share the same metadata, the metadata table needs to have just one row. If they don’t, it needs to hold several rows, one set of metadata for each dataset, identified by a unique primary key (1, 2, …). In order to associate each record of your database with the correct set of metadata, you will need to add a foreign key into the base record table that links to the respective entry in the metadata table. Ask your IT guys if you need help on that. == Controlled Denormalisation == As you will see during configuration, the Provider Software is able to join together the different tables of your database’s data model in order to retrieve all information to be published from the database. This works good if the number of tables is within reasonable limits, that is up to approximately 10-15. Data models used by off-the-self collection management system are usually highly normalised and use far more tables. Specify’s data model, for example, has about 70 tables. Configuring such a complex model in BioCASe will be a terrible pain; moreover, in case your database holds a huge number of records, the time required by the Provider Software for answering search request will become too long. Therefore it is advisable in such cases to add some controlled denormalisation to your database, that is to export the data to be published into several cache tables. These tables could be stored in the same database as the source tables or separate in an extra cache database, maybe even on a different server. They could be created in a regular interval (e.g. each night) or upon request by SQL scripts. The figure below shows this denormalisation process for the data model of Specify. On the left side, you see some of Specify’s source tables (only the ones starting with the letters A, B, C and D). With an SQL script the will be reduced to 7 cache tables, one of them being abcd_object. These tables hold only the columns that will be published, drawn from up to 15 source tables. [[image:prepDenormalisation.png|750px]] == Repeatable Elements in ABCD == One of the main features of ABCD (the schema you will probably use to publish your data) is repeatable elements. For example, ABCD can store multiple identifications or several URLs for multimedia objects attached to a specimen record. All these URLs will be stored in the same ABCD element – namely ''MultimediaObject/FileURI''. Another example is the higher taxonomy (all taxa above genus level). Typically these are stored in different columns in databases (see figure below). In ABCD, they will all end up in the same element, ''HigherTaxon/Name''. In order to be able to map the different ranks stored in columns to the same ABCD concept, we need to pivot these columns into rows – each rank kept in a column needs to be turned into a row, with two columns storing taxon name and rank, one column for linking to the according specimen record, plus one newly created primary key: [[File:prepPivoting.png]] Such a transformation can be easily done with a database view. Views do exactly what the name suggests – they provide a different view on the data. They can be used like regular tables, but they merely represent a different representation of another table. Updates of the underlying tables (or insert/deletions) will also affect views created on these tables, since the data is stored only once in the database. Let’s assume you have the higher taxonomy stored in the specimen table like this: [[File:prepEditTable.png]] The SQL statement <pre> CREATE VIEW [dbo].[vwHigherTaxa] AS SELECT 'k_' + [EDIT_ATBI_RecordID] AS id, [EDIT_ATBI_RecordID] AS unit_id, [kingdom] AS name, 'kingdom' AS rank FROM unit_data WHERE [kingdom] IS NOT NULL UNION ALL SELECT 'p_' + [EDIT_ATBI_RecordID], [EDIT_ATBI_RecordID], [phylum], 'phylum'‚ FROM unit_data WHERE [phylum] IS NOT NULL UNION ALL ... </pre> will create a view that meets the requirements described above. Remember that even though it looks like a table, it’s just another representation of the data stored in the table unit_data: [[File:prepEditView.png]] Such pivoting must be done in all cases when several columns need to be mapped to the same (repeatable) ABCD element. If it’s just one value to be mapped, there is no need for that. Commonly used ABCD elements that are repeatable and might ask for such preparatory transformations are * Identification * HigherTaxon * GatheringSite/NamedArea * Metadata/Scope/GeoecologicalTerms * Metadata/Scope/TaxonomicTerms * MultimediaObjects * MeasurementsOrFacts If you have problems preparing your database for publication, please [[contact the BioCASe team]]. c0a1e48845dabf2a2cdbef00404290df90b6cf6e 171 170 2011-07-21T13:29:01Z JoergHoletschek 3 /* Controlled Denormalisation */ wikitext text/x-wiki Before you start publishing your database with BioCASe, you need to consider several issues. In some cases, some preparation of your database for the publication might be necessary. You should read this page of the wiki careful before you start configuring the Provider Software for a given database. == Live-DB versus Snapshot == The easiest way is to connect BioCASe to the same database you’re working with. This would result in any changes you do being visible on the BioCASe access point right away – which is an idea you might not like. Some curators prefer to publish defined versions of their database, for example after they’ve finished reviewing all specimens of a certain genus. If you prefer this, you need to create a second database that holds defined snapshots of your working database and that will be published instead. Snapshots could be generated in a regular interval (e.g., each Friday evening) or upon request. Please contact your IT guys for more help on this. == Access Control == The BioCASe Provider Software will only publish your data to biodiversity networks, meaning it will only read from the database and never do any changes. For security reasons it’s a good idea to create a user for BioCASe that has only read privileges on the database – more precisely on the tables you want to publish. Since the configuration tool will try to load a list of tables and columns from the database to allow easier setup, this user should also be allowed to load meta-information (<small>SHOW TABLES, SHOW COLUMNS</small>). == Metadata == Metadata describe your dataset as a whole - title, description, curator’s name, ownership, date of last modification, terms of use etc. BioCASe can publish a lot of metadata – and you should take the time to provide as much information as you can, since this will increase the usability of your published data. For storing metadata, please create an extra table in your database and fill it in. If you go to the BioCASe folder of your installation, you’ll find the template for this table in ''sampledb/abcdmetadata''. Either use the SQL script to create it in your database, or import it from the Access database (''metadata.mdb''). Per default, all records of your database will make up one dataset, being described by one set of metadata. In case your database holds records of several datasets, you can also group these into several datasets in ABCD by providing different sets of metadata. So when you realize during gathering of meta-information that they’re not the same for all records, this is an indicator that your database must be grouped into several datasets. If all records to be published share the same metadata, the metadata table needs to have just one row. If they don’t, it needs to hold several rows, one set of metadata for each dataset, identified by a unique primary key (1, 2, …). In order to associate each record of your database with the correct set of metadata, you will need to add a foreign key into the base record table that links to the respective entry in the metadata table. Ask your IT guys if you need help on that. == Controlled Denormalisation == As you will see during configuration, the Provider Software is able to join together the different tables of your database’s data model in order to retrieve all information to be published from the database. This works good if the number of tables is within reasonable limits, that is up to approximately 10-15. Data models used by off-the-shelf collection management system are usually highly normalised and use far more tables. Specify’s data model, for example, has about 70 tables. Configuring such a complex model in BioCASe will be a terrible pain; moreover, in case your database holds a huge number of records, the time required by the Provider Software for answering search request will become too long. Therefore it is advisable in such cases to add some controlled denormalisation to your database, that is to export the data to be published into several cache tables. These tables could be stored in the same database as the source tables or separate in an extra cache database, maybe even on a different server. They could be created in a regular interval (e.g. each night) or upon request by SQL scripts. The figure below shows this denormalisation process for the data model of Specify. On the left side, you see some of Specify’s source tables (only the ones starting with the letters A, B, C and D). With an SQL script they will be reduced to 7 cache tables, one of them being abcd_object. These tables hold only the columns that will be published, drawn from up to 15 source tables. [[image:prepDenormalisation.png|750px]] == Repeatable Elements in ABCD == One of the main features of ABCD (the schema you will probably use to publish your data) is repeatable elements. For example, ABCD can store multiple identifications or several URLs for multimedia objects attached to a specimen record. All these URLs will be stored in the same ABCD element – namely ''MultimediaObject/FileURI''. Another example is the higher taxonomy (all taxa above genus level). Typically these are stored in different columns in databases (see figure below). In ABCD, they will all end up in the same element, ''HigherTaxon/Name''. In order to be able to map the different ranks stored in columns to the same ABCD concept, we need to pivot these columns into rows – each rank kept in a column needs to be turned into a row, with two columns storing taxon name and rank, one column for linking to the according specimen record, plus one newly created primary key: [[File:prepPivoting.png]] Such a transformation can be easily done with a database view. Views do exactly what the name suggests – they provide a different view on the data. They can be used like regular tables, but they merely represent a different representation of another table. Updates of the underlying tables (or insert/deletions) will also affect views created on these tables, since the data is stored only once in the database. Let’s assume you have the higher taxonomy stored in the specimen table like this: [[File:prepEditTable.png]] The SQL statement <pre> CREATE VIEW [dbo].[vwHigherTaxa] AS SELECT 'k_' + [EDIT_ATBI_RecordID] AS id, [EDIT_ATBI_RecordID] AS unit_id, [kingdom] AS name, 'kingdom' AS rank FROM unit_data WHERE [kingdom] IS NOT NULL UNION ALL SELECT 'p_' + [EDIT_ATBI_RecordID], [EDIT_ATBI_RecordID], [phylum], 'phylum'‚ FROM unit_data WHERE [phylum] IS NOT NULL UNION ALL ... </pre> will create a view that meets the requirements described above. Remember that even though it looks like a table, it’s just another representation of the data stored in the table unit_data: [[File:prepEditView.png]] Such pivoting must be done in all cases when several columns need to be mapped to the same (repeatable) ABCD element. If it’s just one value to be mapped, there is no need for that. Commonly used ABCD elements that are repeatable and might ask for such preparatory transformations are * Identification * HigherTaxon * GatheringSite/NamedArea * Metadata/Scope/GeoecologicalTerms * Metadata/Scope/TaxonomicTerms * MultimediaObjects * MeasurementsOrFacts If you have problems preparing your database for publication, please [[contact the BioCASe team]]. cfdce5a88bc967cf60c8bd0c1448b9a8b2363ff1 6 56 173 2011-07-21T13:43:28Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 57 177 2011-07-21T13:55:30Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 8 3 184 3 2011-07-21T14:53:08Z JoergHoletschek 3 wikitext text/x-wiki * SEARCH * navigation ** mainpage|mainpage-description ** portal-url|portal ** currentevents-url|currentevents ** recentchanges-url|recentchanges ** randompage-url|randompage ** mediawikiwiki:Help:Contents|help * TOOLBOX * LANGUAGES 7d3ff6f052916aa7c5094baa9cbec59303c60d99 185 184 2011-07-21T14:56:54Z JoergHoletschek 3 wikitext text/x-wiki * navigation ** mainpage|mainpage-description ** portal-url|portal ** currentevents-url|currentevents ** recentchanges-url|recentchanges ** randompage-url|randompage ** mediawikiwiki:Help:Contents|help * SEARCH * TOOLBOX * LANGUAGES 3815dfdeba7a9affe9eeda9aa612a0c227e89ce5 186 185 2011-07-21T14:59:31Z JoergHoletschek 3 wikitext text/x-wiki * navigation ** mainpage|mainpage-description ** portal-url|portal ** currentevents-url|currentevents ** recentchanges-url|recentchanges ** randompage-url|randompage ** mediawikiwiki:Help:Contents|help * SEARCH * TOOLBOX * LANGUAGES f6bd0c797f209a03a36f251edd3ee8cd3888107f 187 186 2011-07-21T15:02:03Z JoergHoletschek 3 wikitext text/x-wiki * navigation ** BeginnersGuide|Beginner's Guide ;[[Preparation]]: Things to be considered when publishing a database with BioCASe and potential preparatory steps needed. [[Image:support_stamp.png|link=Contact_the_BioCASe_team|200px|right|BioCASe Support]] ;[[Installation]]: Tutorial on how to install the BioCASe Provider Software. ;[[DatasourceSetup|Datasource Setup]]: How to set up a new BioCASe web service (aka data source). ;[[ABCD2Mapping|ABCD2 Mapping]]: Tutorial on mapping ABCD2. ;[[CommonABCD2Concepts|Common ABCD2 Concepts]]: List of mandatory, recommended and commonly used ABCD2 elements. ;[[SampleABCDDocument|Sample ABCD2 Document]]: Example ABCD2 document of a comprehensively mapped data provider. ;[[Debugging|Debugging Tutorial]]: Instructions on how to debug a BioCASe web service. ;[[Glossary|Glossary]]: List of terms and abbreviations used in this documentation. ;[[FAQ|FAQ]]: List of frequently asked questions and hopefully helpful answers. ;[[VersionHistory|Version History]]: Change list of current and past versions of the BioCASe Provider Software. ** mainpage|mainpage-description ** portal-url|portal ** currentevents-url|currentevents ** recentchanges-url|recentchanges ** randompage-url|randompage ** mediawikiwiki:Help:Contents|help * SEARCH * TOOLBOX * LANGUAGES 3f7cde89c96e92ecc22eec12b9d8c5085121ee43 188 187 2011-07-21T15:02:53Z JoergHoletschek 3 wikitext text/x-wiki * navigation ** mainpage|mainpage-description ** BeginnersGuide|Beginner's Guide ** Preparation [[Image:support_stamp.png|link=Contact_the_BioCASe_team|200px|right|BioCASe Support]] ;[[Installation]]: Tutorial on how to install the BioCASe Provider Software. ;[[DatasourceSetup|Datasource Setup]]: How to set up a new BioCASe web service (aka data source). ;[[ABCD2Mapping|ABCD2 Mapping]]: Tutorial on mapping ABCD2. ;[[CommonABCD2Concepts|Common ABCD2 Concepts]]: List of mandatory, recommended and commonly used ABCD2 elements. ;[[SampleABCDDocument|Sample ABCD2 Document]]: Example ABCD2 document of a comprehensively mapped data provider. ;[[Debugging|Debugging Tutorial]]: Instructions on how to debug a BioCASe web service. ;[[Glossary|Glossary]]: List of terms and abbreviations used in this documentation. ;[[FAQ|FAQ]]: List of frequently asked questions and hopefully helpful answers. ;[[VersionHistory|Version History]]: Change list of current and past versions of the BioCASe Provider Software. ** portal-url|portal ** currentevents-url|currentevents ** recentchanges-url|recentchanges ** randompage-url|randompage ** mediawikiwiki:Help:Contents|help * SEARCH * TOOLBOX * LANGUAGES 5f4927a8fda9c93c377069cd466391c810b3e09d 189 188 2011-07-21T15:04:40Z JoergHoletschek 3 wikitext text/x-wiki * navigation ** mainpage|mainpage-description ** BeginnersGuide|Beginner's Guide ** Preparation|Preparation ** Installation|Installation ** DatasourceSetup|Datasource Setup ** ABCD2Mapping|ABCD2 Mapping ** ABCD2Concepts|ABCD2 Concepts ** SampleABCDDocument|Sample ABCD2 Document ** Debugging|Debugging ** Glossary|Glossary ** FAQ|FAQ ** VersionHistory|Version History ** portal-url|portal ** currentevents-url|currentevents ** recentchanges-url|recentchanges ** randompage-url|randompage ** mediawikiwiki:Help:Contents|help * SEARCH * TOOLBOX * LANGUAGES 10f9e6804f22326fa240aac5762347a995a886cf 190 189 2011-07-21T15:05:06Z JoergHoletschek 3 wikitext text/x-wiki * navigation ** mainpage|mainpage-description ** BeginnersGuide|Beginner's Guide ** Preparation|Preparation ** Installation|Installation ** DatasourceSetup|Datasource Setup ** ABCD2Mapping|ABCD2 Mapping ** ABCD2Concepts|ABCD2 Concepts ** SampleABCDDocument|ABCD2 Sample ** Debugging|Debugging ** Glossary|Glossary ** FAQ|FAQ ** VersionHistory|Version History ** portal-url|portal ** currentevents-url|currentevents ** recentchanges-url|recentchanges ** randompage-url|randompage ** mediawikiwiki:Help:Contents|help * SEARCH * TOOLBOX * LANGUAGES 0dd071947989e492db951cc534727a17d3a0314b 191 190 2011-07-21T15:05:54Z JoergHoletschek 3 wikitext text/x-wiki * navigation ** mainpage|mainpage-description ** BeginnersGuide|Beginner's Guide ** Preparation|Preparation ** Installation|Installation ** DatasourceSetup|Datasource Setup ** ABCD2Mapping|ABCD2 Mapping ** Debugging|Debugging ** Glossary|Glossary ** FAQ|FAQ * SEARCH * TOOLBOX d4715803e78b038f37cfc6a83d8ef1453d7f993d 192 191 2011-07-21T15:13:13Z JoergHoletschek 3 wikitext text/x-wiki * Table of Contents ** mainpage|mainpage-description ** BeginnersGuide|Beginner's Guide ** Preparation|Preparation ** Installation|Installation ** DatasourceSetup|Datasource Setup ** ABCD2Mapping|ABCD2 Mapping ** Debugging|Debugging ** Glossary|Glossary ** FAQ|FAQ * SEARCH * TOOLBOX 0852cbb5c7a95583081d8676641ff7d43ac40a65 193 192 2011-07-21T15:14:24Z JoergHoletschek 3 wikitext text/x-wiki * mainpage|mainpage-description * Table of Contents ** BeginnersGuide|Beginner's Guide ** Preparation|Preparation ** Installation|Installation ** DatasourceSetup|Datasource Setup ** ABCD2Mapping|ABCD2 Mapping ** Debugging|Debugging ** Glossary|Glossary ** FAQ|FAQ * SEARCH * TOOLBOX ec05eba8a4add3dd412aee45efdfe6792ae52d0d 194 193 2011-07-21T15:15:10Z JoergHoletschek 3 wikitext text/x-wiki * Main_Page|Main Page * Table of Contents * mainpage|mainpage-description ** BeginnersGuide|Beginner's Guide ** Preparation|Preparation ** Installation|Installation ** DatasourceSetup|Datasource Setup ** ABCD2Mapping|ABCD2 Mapping ** Debugging|Debugging ** Glossary|Glossary ** FAQ|FAQ * SEARCH * TOOLBOX e7dc66b5cdd87e8bd26c01c8b726f98cd40d2abc 195 194 2011-07-21T15:16:41Z JoergHoletschek 3 wikitext text/x-wiki * Table of Contents ** mainpage|mainpage-description ** BeginnersGuide|Beginner's Guide ** Preparation|Preparation ** Installation|Installation ** DatasourceSetup|Datasource Setup ** ABCD2Mapping|ABCD2 Mapping ** Debugging|Debugging ** Glossary|Glossary ** FAQ|FAQ * SEARCH * TOOLBOX a9793b707477e2bae75b99156c84b3e443221fb2 196 195 2011-07-21T15:17:43Z JoergHoletschek 3 wikitext text/x-wiki * Table of Contents ** mainpage|mainpage-description ** BeginnersGuide|Beginner's Guide ** Preparation|Preparation ** Installation|Installation ** DatasourceSetup|Datasource Setup ** ABCD2Mapping|ABCD2 Mapping ** Debugging|Debugging ** Glossary|Glossary ** FAQ|FAQ * Contact ** Comments|Comments Page ** Contact_the_BioCASe_team|Contact us * SEARCH * TOOLBOX a90e438f48be829db05ed29519f67f958096d453 197 196 2011-07-21T15:21:17Z JoergHoletschek 3 wikitext text/x-wiki * Table of Contents ** mainpage|mainpage-description ** BeginnersGuide|Beginner's Guide ** Preparation|Preparation ** Installation|Installation ** DatasourceSetup|Datasource Setup ** ABCD2Mapping|ABCD2 Mapping ** Debugging|Debugging ** Glossary|Glossary ** FAQ|FAQ * ABCD 2.06 ** CommonABCD2Concepts|Common Elements ** SampleABCDDocument|Sample Document * Contact ** Comments|Comments Page ** Contact_the_BioCASe_team|Contact us * SEARCH * TOOLBOX 69fa95607248b31cbc857912ec4aaa8bfd253c6a 199 197 2011-07-21T15:22:49Z JoergHoletschek 3 wikitext text/x-wiki * Table of Contents ** mainpage|mainpage-description ** BeginnersGuide|Beginner's Guide ** Preparation|Preparation ** Installation|Installation ** DatasourceSetup|Datasource Setup ** ABCD2Mapping|ABCD2 Mapping ** Debugging|Debugging ** Glossary|Glossary ** FAQ|FAQ * ABCD 2.06 ** CommonABCD2Concepts|Common Elements ** SampleABCDDocument|Sample Document * Feedback ** Comments|Comments Page ** Contact_the_BioCASe_team|Contact us * SEARCH * TOOLBOX 184922ff308e48613577fd83e16b3d5e6ff9a805 201 199 2011-07-21T15:24:47Z JoergHoletschek 3 wikitext text/x-wiki * Table of Contents ** mainpage|mainpage-description ** BeginnersGuide|Beginner's Guide ** Preparation|Preparation ** Installation|Installation ** DatasourceSetup|Datasource Setup ** ABCD2Mapping|ABCD Mapping ** Debugging|Debugging ** Glossary|Glossary ** FAQ|FAQ * ABCD 2.06 ** CommonABCD2Concepts|Common Elements ** SampleABCDDocument|Sample Document * Feedback ** Comments|Comments Page ** Contact_the_BioCASe_team|Contact us * SEARCH * TOOLBOX e9fed23395b830a9699dafbb4a26f15731f96f1d 202 201 2011-07-21T15:55:06Z JoergHoletschek 3 wikitext text/x-wiki * Table of Contents ** mainpage|mainpage-description ** BeginnersGuide|Beginner's Guide ** Preparation|Preparation ** Installation|Installation ** DatasourceSetup|Datasource Setup ** ABCD2Mapping|ABCD Mapping ** Debugging|Debugging ** Glossary|Glossary ** FAQ|FAQ * ABCD 2.06 ** CommonABCD2Concepts|Common Elements ** SampleABCDDocument|Sample Document * Feedback ** Comments|Comments Page ** Contact_the_BioCASe_team|Contact us * SEARCH 868d6da8f6e26e1c231c07a8672b8bd1bee24132 203 202 2011-07-21T15:55:32Z JoergHoletschek 3 wikitext text/x-wiki * Table of Contents ** mainpage|mainpage-description ** BeginnersGuide|Beginner's Guide ** Preparation|Preparation ** Installation|Installation ** DatasourceSetup|Datasource Setup ** ABCD2Mapping|ABCD Mapping ** Debugging|Debugging ** Glossary|Glossary ** FAQ|FAQ * ABCD 2.06 ** CommonABCD2Concepts|Common Elements ** SampleABCDDocument|Sample Document * Feedback ** Comments|Comments Page ** Contact_the_BioCASe_team|Contact us a668c340fe74afc2088e976516b822b0f5de59e9 0 49 198 135 2011-07-21T15:22:04Z JoergHoletschek 3 /* Steps */ wikitext text/x-wiki == What is the BioCASe Provider Software? == The BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. The Provider Software is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. The Provider Software is a suite of tools that need to be installed on the data provider’s web server. The core component is the PyWrapper, an XML/CGI database interface written in Python that allows a standardized access to a variety of database management systems and arbitrarily structured databases. Grouped around this are a number of tools for configuring, testing and debugging a BioCASe installation or web service. Even though BioCASe can be used for any conceptual XML data schema, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the [http://www.biocase.org BioCASe network] and the [http://www.gbif.org Global Biodiversity Information Facility]. == Requirements for Using the BPS == Before using the BioCASe Provider Software, two requirements need to be checked: * For one thing, the data to be published needs to be stored in an SQL compliant relational database management system (DBMS), with the data model being well documented or at least knowledge about the data model existing on someone’s head. Non-SQL DBMS are not supported. Currently, the BioCASe Provider Software can connect to the following databases: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * For installing BioCASe, you need to have a web server (Apache or Microsoft IIS) that is permanently connected to the Internet. This can be an institutional server of a hosting service (even though it’s usually a pain to get privileges to install software for the latter). Dial-up connections won’t work. == Steps == Once you’ve made sure the BioCASe Provider Software is the right choice for you ([[contact the BioCASe team]] if you feel unsafe with your decision), you need to do the following steps: # Think about which information you want to publish to the network. This involves settling the question of ownership of the data and the terms of use, license and usage restrictions for the published data. Also you should consider which information should '''not''' be published because they might affect endangered species (options are to block certain occurrences from publication or to blur the exact locality information). # Prepare the database for publication as described in the [[Preparation]] tutorial. This includes the decision on whether to publish the live database or a snapshot, the creation of a table for metadata and, in certain cases, some data transformation. # Follow the [[Installation]] Tutorial to install the BioCASe Provider Software. # Set up a BioCASe Datasource and connect it to your database, then configure the database structure as described in the [[DatasourceSetup]] tutorial. # Follow the [[ABCD2Mapping]] tutorial to create a mapping for the schema you want to use. # Test and debug your BioCASe web service. Read the [[Debugging]] tutorial to learn how to do this. # Once you’re done with setting up and configuring the BioCASe Provider Software, you can register the BioCASe web service with the information network you want to publish your data to, for example with [http://www.gbif.org GBIF]. We strongly recommend to [[contact the BioCASe team]] for checking your web service first as we might be able to foresee problems that are more difficult to resolve after registration. == Support == In case you run into difficulties during the installation and setup, you should * have a look at the FAQ which might list the solution for your question, * ask other BioCASe data providers for help, * get in contact with the [[contact the BioCASe team|BioCASe team]]. Please describe exactly you environment (operating system, DBMS used), the steps that cause the problem and any error messages and hints you get. Screenshot are a big plus here. ed814302e6923aa8e8e0d0b4322349ba273abdc4 205 198 2011-08-01T09:05:47Z JoergHoletschek 3 /* Support */ wikitext text/x-wiki == What is the BioCASe Provider Software? == The BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. The Provider Software is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. The Provider Software is a suite of tools that need to be installed on the data provider’s web server. The core component is the PyWrapper, an XML/CGI database interface written in Python that allows a standardized access to a variety of database management systems and arbitrarily structured databases. Grouped around this are a number of tools for configuring, testing and debugging a BioCASe installation or web service. Even though BioCASe can be used for any conceptual XML data schema, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the [http://www.biocase.org BioCASe network] and the [http://www.gbif.org Global Biodiversity Information Facility]. == Requirements for Using the BPS == Before using the BioCASe Provider Software, two requirements need to be checked: * For one thing, the data to be published needs to be stored in an SQL compliant relational database management system (DBMS), with the data model being well documented or at least knowledge about the data model existing on someone’s head. Non-SQL DBMS are not supported. Currently, the BioCASe Provider Software can connect to the following databases: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * For installing BioCASe, you need to have a web server (Apache or Microsoft IIS) that is permanently connected to the Internet. This can be an institutional server of a hosting service (even though it’s usually a pain to get privileges to install software for the latter). Dial-up connections won’t work. == Steps == Once you’ve made sure the BioCASe Provider Software is the right choice for you ([[contact the BioCASe team]] if you feel unsafe with your decision), you need to do the following steps: # Think about which information you want to publish to the network. This involves settling the question of ownership of the data and the terms of use, license and usage restrictions for the published data. Also you should consider which information should '''not''' be published because they might affect endangered species (options are to block certain occurrences from publication or to blur the exact locality information). # Prepare the database for publication as described in the [[Preparation]] tutorial. This includes the decision on whether to publish the live database or a snapshot, the creation of a table for metadata and, in certain cases, some data transformation. # Follow the [[Installation]] Tutorial to install the BioCASe Provider Software. # Set up a BioCASe Datasource and connect it to your database, then configure the database structure as described in the [[DatasourceSetup]] tutorial. # Follow the [[ABCD2Mapping]] tutorial to create a mapping for the schema you want to use. # Test and debug your BioCASe web service. Read the [[Debugging]] tutorial to learn how to do this. # Once you’re done with setting up and configuring the BioCASe Provider Software, you can register the BioCASe web service with the information network you want to publish your data to, for example with [http://www.gbif.org GBIF]. We strongly recommend to [[contact the BioCASe team]] for checking your web service first as we might be able to foresee problems that are more difficult to resolve after registration. == Support == In case you run into difficulties during the installation and setup, you should * have a look at the [[FAQ]] which might list the solution for your question, * ask other BioCASe data providers for help, * get in contact with the [[contact the BioCASe team|BioCASe team]]. Please describe exactly you environment (operating system, DBMS used), the steps that cause the problem and any error messages and hints you get. Screenshot are a big plus here. a629a882cbc042e45dd7740dcf7113675edd191e 0 29 204 142 2011-07-21T16:05:42Z JoergHoletschek 3 /* The Mapping Editor */ wikitext text/x-wiki This tutorial will explain how to set up a mapping for the ABCD2.06 data schema. If you intend to map another schema (e.g. one of the ABCD extension ABCD-EFG, ABCD-DNA or HISPID), you still should take the time to read it. Large portions of the process will be identical. == Access to Biological Collection Data Schema (ABCD) == ABCD in its current version 2.06 is the data schema typically used in conjunction with BioCASe. It is a highly complex XML data schema with about 1,000 elements that is able to store almost every piece of information that can be found in a natural history specimen collection or an observation database. The complexity is due to the fact that it can be used for a wide range of collections/databases – for living and preserved specimens, observations and culture collections, for zoological, botanical, bacterial and viral collections, marine or terrestrial, for herbaria, botanic and zoological gardens. For each of these special types, ABCD features special sections where information specific for this type can be stored. Other sections of ABCD will be shared by all these types, for example for gathering site, identifier or metadata. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a list of commonly used ABCD elements on the Wiki page [[CommonABCD2Concepts]]. == Creating an ABCD Mapping for a Data Source == If you created a data source from a template (e.g. ''abcdmetadata''), a schema mapping for ABCD will be already existent. In this case, you can directly jump to the next section. If you created an empty data source, the section <small>Schemas</small> in the data source configuration overview will be empty: [[File:abcdSchemaMappings.png]] In order to create a new schema mapping, select the desired schema from the list and click <small>Create</small>. For ABCD, choose <small>ABCD_2.06.xml</small> (be careful not to hit one of the ABCD extensions, they look quite similar). After clicking <small>Create</small>, you will be directed to the mapping editor, which will allow you to edit the mappings. == The Mapping Editor == The mapping editor allows you to add, remove and edit ABCD mappings. On top you’ll find some summary information, on the bottom you’ll see the structure of the ABCD mapping: [[File:abcdMappingEditorEmpty.png]] First thing we will do is to set the root table, which is the table that holds the records you want to publish (specimens or observations). That table must hold one record per object to be published, identified by a unique identifier (called ''UnitID''). This identifier doesn't need to be the primary key necessarily, as long as it is unique (if that sentence confuses you, ignore it and just keep in mind that the UnitID must be unique.) Use the drop down box labelled <small>Root table Alias</small> to select the root table of your data model. If your metadata table holds just one record (because all occurrences published share the same metadata) and if it is not linked to the root table by a foreign key, you can specify this table in the drop-down box <small>Static table Alias</small>. The Provider Software will use a natural join to connect the metadata record stored in this table to all occurrences published. If your metadata table is referenced by a foreign key in the root able, you must not declare the metadata table as a static table! So either join the metadata table using a foreign key OR declare it as a static table. When you’ve set the root table (and static table, if you use that), press <small>Save</small>. As with the DB structure setup, this will write your changes to the configuration files, closing the tab without saving will discard your changes! Also, in case you messed something up, you can use the <small>Revert</small> button to restore the state when you’ve saved for the last time. In the mapping editor, the link <small>Overview</small> at the top of the page will take you back to the datasource configuration overview. After creating a mapping, it will appear in the section <small>Schemas</small>. The entry will show the number of mapped elements (0 for now) and the schema namespace. Pressing the trash can symbol will move it to the trash can, which means that your web service does not support this schema anymore; if you did this by mistake, you can restore it by pressing the <small>Restore</small> button. <small>Purge</small> will delete the schema mapping permanently: [[File:abcdSchemaMappings2.png]] You can map several schemas for a given datasource. For example, in addition to the current ABCD2 standard, you could add the deprecated version 1.2 (even though I have no idea why you should do that) or DarwinCore. Just select the appropriate schema in the drop-down list and click <small>Create</small>. Since the namespace of a schema will be used in requests, you cannot create two mappings for the same namespace. Trying this will get you an error message. == Adding Mandatory ABCD Elements == Clicking on a schema in the datasource overview page will take you again to the mapping editor. In the lower half of the page, you will see the basic structure of an ABCD document: [[File:abcdMappingEditorEmptyBottom.png]] <strong>Technical/Content Contact:</strong> These trees hold information on who should be contacted for technical issues or questions concerning the published database, each with name, address, email address and telephone number. <strong>Metadata:</strong> In this sub tree you’ll find all metadata fields, for example title and description of the dataset, ownership information, date last modified, IPR statements etc. <strong>Units:</strong> This part stores the "real" data you want to publish, that means all specimen or observation data. The three elements already shown in the tree are identifiers that make up the so called "Triple ID", which uniquely identifies records within primary biodiversity networks such as BioCASe and GBIF: * ''UnitID'', which is a unique identifier for each record to be published (you can use the catalogue number, for example, or an existing barcode); * ''SourceInstitutionID'', an identifier for the institution publishing the dataset (for example BGBM); * ''SourceID'', an identifier for the dataset (e.g. “Herbarium Berolinense’’). The elements already shown in the mapping editor are printed in red because they are mandatory, which means they are required for an ABCD document. Without them being mapped (to database fields that are non-empty), the Provider Software will not be able to construct valid ABCD documents and will not publish any records. So the first elements that should be mapped are the mandatory fields. To add a mapping, click the <strong>+</strong> symbol next to the field, which will open the mapping editor dialog, where you can select the database table/column you want to map the ABCD element to: [[File:abcdMappingEditorDialog.png]] In this case, the element ''/DataSets/DataSet/Metadata/RevisionData/DateModified'' is mapped to the column ''source_update'' of the ''metadata'' table, which has the data type ''date''. (Remember that table/column retrieval is not supported for all DBMS, so if you’re unlucky, you might need to type in table/column name manually.) <small>Save</small> will accept your settings and close the dialog, <small>Cancel</small> will discard any changes. Instead of mapping an element to a database field, you can map it to a literal. For example, you could enter the technical and content contact information directly as literals in the mapping editor dialog (assuming they’re the same for all records). However, we strongly discourage you from doing this. For one thing, this is just bad style, since it will scatter the information published over different places: Some will be retrieved from the database, some (the literals) will be loaded from the configuration files. Changing the metadata of your service later would then require access to the database and access to the BioCASe configuration – that means the configuration tool password. Be nice to the person who will do these changes and put everything neatly in a database table! The second reason is even more important: Using literals for an element will result in this element to become not searchable, which means it cannot be used in a filter of a Search or a Scan request. Even though this behaviour is discoverable with a Capabilities request (by evaluating the ''searchable'' flag), a potential user of your web service might just not think of that, so it’s a good idea to avoid that in first place. Please use literals sparsely! In the mapping editor window, you can select several source columns and several literals, even a combination of both. The Provider Software will simply concatenate the values retrieved from the database and the literals specified (in the order they’re given). But again: Keep in mind that this will make the element not searchable. A better solution is to do any concatenation operations in a database view. Use the mapping editor window to add mappings for all eight mandatory fields: Click on the <strong>+</strong> symbol next to the element, choose table/column name and the correct data type, then press <small>Save</small>. When you’re done, the page should look similar to the following image. Press <small>Save</small> in the main window to write your changes to the configuration files. [[File:abcdMappingEditorMandatoryElements.png]] == Initial Testing of the Data Source == Once you’ve mapped all mandatory elements, you can do the first test of your BioCASe web service. Even though you haven’t mapped any real information yet, it is a good idea to check the service and solve any problems hitherto before continuing with the mapping. Click on <small>Test mapping!</small> to open the Query Form that can be used for this purpose: [[File:abcdQueryFormEmpty.png]] With the Query Form you can send a BioCASe request to your web service; it will wait for the reply of the web service and display the response document. On top of the page you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ''flora'' being the name of the datasource. Make sure this is correct for your installation! Into the text box below you can enter the request XML text. Because it would be cumbersome to enter the whole BioCASe request, you can use the links below the box to load templates. For now, click on <small>ABCD2 Search</small> to fill the box with a BioCASe Search request for documents in the ABCD2 format (the one we’ve just mapped): <pre> <?xml version='1.0' encoding='UTF-8'?> <request xmlns='http://www.biocase.org/schemas/protocol/1.3'> <header><type>search</type></header> <search> <requestFormat>http://www.tdwg.org/schemas/abcd/2.06</requestFormat> <responseFormat start='0' limit='10'>http://www.tdwg.org/schemas/abcd/2.06</responseFormat> <filter> <like path='/DataSets/DataSet/Units/Unit/Identifications/Identification/Result/ TaxonIdentified/ScientificName/FullScientificNameString'>A*</like> </filter> <count>false</count> </search> </request> </pre> As you can see, there is a filter set on the ''ScientificName'' element. Since we haven’t mapped this element yet, this won’t work. So let’s remove the filter (all lines between <filter> and </filter>), leaving the text box like this: [[File:abcdQueryFormFilled.png]] Pressing <small>Submit</small> will send the request to the web service and – hopefully – display the response document with the correct ABCD records. For the mapping we’ve just created, it should look similar to the following document (use the <strong>–</strong> symbol next to the trees TechnicalContact, ContentContact and Metadata to collapse them): [[File:abcdResponseMandatoryElements.png]] If you get an error, please check out the [[Debugging]] tutorial on how to debug a web service. You should continue the mapping process only after your web service returns results for the basal mapping we’ve just created. == Adding additional ABCD Elements == Once you’ve successfully managed to map the mandatory ABCD elements ("successfully" means the web service returns ABCD documents), you can go on with the mapping process. This is usually an iterative process: You add a handful of elements (for example, country, country code, locality text and coordinates for the gathering site), then you use the Query Form to see if the information ends up where it is supposed to do. This is the reason the Query Form opens up in a separate tab (or window, of you configured your web browser so): In one tab, you use the mapping editor to add new elements; after pressing <small>Save</small> you go to the Query Form tab and see what the response documents look like. Redo this until you’ve mapped all information you want to publish. In order to map new ABCD concepts, tick the check box labelled <small>Show all concepts</small> and press <small>Refresh</small>. The mapping editor will then display the whole ABCD tree, which looks admittedly a bit intimidating. In order to find the concept you’re looking for, use the search function of your browser. If you want to map the country name of the gathering site, for example, hit <small>Ctrl + F</small> (for Firefox, Microsoft IE and Opera) and type in the name of the ABCD element, namely ''country''. The browser will jump to the ''Country'' tree of the ''Gathering'' element, where you’ll find the elements for country name (''Name''), ISO country code (''ISO3166Code'') and the country name in the country’s language (''NameDerived''): [[File:abcdMappingEditorSearch.png|500px]] Continue the sequence mapping new elements/testing until you’ve mapped all information you want to publish. You can have a look at the list of recommend and commonly used ABCD elements on the [[CommonABCD2Concepts]] page. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki]. The following page will show you a sample response ABCD document for a web service with 77 concepts mapped. Of course your documents will look different, because you will probably use different ABCD elements, but still it’s good to get an idea. For easier viewing, you can save the contents of the box into a file with the extension .xml and open that in Firefix or MS Internet Explorer (don't use Safari). You will then be able to collapse the different XML sub trees: [[SampleABCDDocument]]. 22c8980762b1b53bb1bec7ae8a1461a11b087c61 0 52 206 158 2011-08-01T09:09:06Z JoergHoletschek 3 /* General */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===When can I use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just contact us and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I’ve forgot the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might just found a bag. Please take a screenshot of the screen and write down what you did to make this happen. Then [[contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[contact the BioCASe team|contact us]]; we’ll try our best to help you. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the [[Comments]] page or get directly in [[contact the BioCASe team|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry (you might need to close and reopen your browser). If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I cannot find the installation package for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). a94b2eea49595b818ba8403e1c1ee18263ea7f66 207 206 2011-08-02T08:52:13Z JoergHoletschek 3 /* Debugging */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===When can I use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just contact us and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I’ve forgot the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might just found a bag. Please take a screenshot of the screen and write down what you did to make this happen. Then [[contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[contact the BioCASe team|contact us]]; we’ll try our best to help you. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the [[Comments]] page or get directly in [[contact the BioCASe team|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry (you might need to close and reopen your browser). If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I cannot find the installation package for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. b4afa92b1cd85ac4f2e3f2d96a7da55906d3d1c2 209 207 2011-08-04T08:37:50Z JoergHoletschek 3 /* Installation */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===When can I use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just contact us and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I’ve forgot the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might just found a bag. Please take a screenshot of the screen and write down what you did to make this happen. Then [[contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[contact the BioCASe team|contact us]]; we’ll try our best to help you. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the [[Comments]] page or get directly in [[contact the BioCASe team|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry (you might need to close and reopen your browser). If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I cannot find the installation package for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. 42b9a70b8ffeb3123ddb24bfcd1c21ab4344b2fd 0 23 208 138 2011-08-03T16:27:01Z JoergHoletschek 3 /* Version 2.6.1 [upcoming] */ wikitext text/x-wiki == Version 2 History == ===Version 2.6.1 [upcoming]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.5.5 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. See example [http://ww3.bgbm.org/bps2/ScreenShots?action=AttachFile&do=get&target=config_tool4.gif here] * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 23974f51a727b7e9797c44d199a5df723c807f1c 0 6 210 143 2011-08-05T11:36:12Z 192.168.2.133 0 small test wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Contact the BioCASe team|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in (in case Python is not found, you’ll have to add the full path name to the Python executable):<pre>python setup.py</pre>Make sure the Python version given at the beginning out the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] All of the questions do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Import configuration from existing installation:</strong> Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.foobar.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.foobar.org/biocase'', for example. After answering these three questions, the script will update your files. At the end of the output it will print the lines you will need later to create the link, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so will compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.foobar.org/biocase''. If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can a pain; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the DBMS used by the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to he library test page that shows the status of installed libraries. On a vanilla BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries ship with the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' in the browser window), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown [[Installation#Linking_BioCASe_and_Your_Web_Server|here]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] 4a5a42effcba7ba6f0d6f671a4b06ca6b8813ecc 1 58 211 2011-08-05T11:38:11Z 192.168.2.133 0 page created wikitext text/x-wiki If you have questions, post them here. 515816eb14ac3ced1b2c5937c02cc2fa777e9378 238 211 2011-08-09T11:56:18Z JoergHoletschek 3 wikitext text/x-wiki {{Template:Discussion}} <!--Please add your comment below this line. --> -------------------------------------------------------------------------------- ;Short Description of your comment: Please write your comment here. 437b5ff4194e1283c0fc47af38be723a243b420e 8 2 212 2 2011-08-08T15:52:49Z JoergHoletschek 3 css text/css /* CSS placed here will be applied to all skins */ /* Hide title of Main_Page */ /* body.page-Main_Page h1.firstHeading { display:none; } */ /* Hide every title except special pages */ body.ns-subject h1.firstHeading { display:none; } /* Hide toolbox */ #p-tb{ display:none;} b8feaf58373c830b05ad288c4bf18bdc6ee0b406 213 212 2011-08-08T15:54:13Z JoergHoletschek 3 css text/css /* CSS placed here will be applied to all skins */ /* Hide title of Main_Page */ /* body.page-Main_Page h1.firstHeading { display:none; } */ /* Hide every title except special pages */ /*body.ns-subject h1.firstHeading { display:none; }*/ /* Hide toolbox */ #p-tb{ display:none;} ba699c7a3b60f8587787997b070867f28568ed16 214 213 2011-08-08T16:16:28Z JoergHoletschek 3 css text/css /* CSS placed here will be applied to all skins */ /* Hide title of Main_Page */ /* body.page-Main_Page h1.firstHeading { display:none; } */ /* Hide every title except special pages */ /*body.ns-subject h1.firstHeading { display:none; }*/ /* Added manually: Hide toolbox */ #p-tb{ display:none;} /* Added manually: Move BreadcrumbsTrail below Heading */ #BreadCrumbsTrail { margin-bottom: -30px; position: relative; top: -70px; } #firstHeading { margin-top: 20px; } b9178525d6b01416c47ec5281a3393c4070e965c 0 1 215 200 2011-08-09T10:25:27Z JoergHoletschek 3 wikitext text/x-wiki This is the Documentation Wiki of the BioCASe Provider Software. Here you will find tutorials on how to install, configure and debug the BioCASe Provider Software. More detailed information on BioCASe can be found in the [[BeginnersGuide|Beginner's Guide]] and on the [http://www.biocase.org/whats_biocase BioCASe homepage]. [[Image:BioCASeLogoLarge.jpg|link=http://www.biocase.org|150px|right|BioCASe]]Described in one paragraph, the BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. It is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. However, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the BioCASe network (http://www.biocase.org) and the Global Biodiversity Information Facility (http://www.gbif.org). After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. == BioCASe Support == [[Image:mail.png|link=Contact_the_BioCASe_team|150px|right|BioCASe Support]] You can use the [[Comments|Comments page]] to just leave us a comment. If you want us to reply to your feedback, you should [[Contact_the_BioCASe_team|get in contact with the BioCASe Team]] directly. Please do so if you * have any questions concerning BioCASe in general or the installation/configuration of the Provider Software, * want to file a bug report, * have any comments, or * want to make a suggestion for a useful new feature in upcoming versions. == Table of Contents == ;[[BeginnersGuide|Beginner's Guide]]: What the Provider Software is all about. ;[[Preparation]]: Things to be considered when publishing a database with BioCASe and potential preparatory steps needed. [[Image:support_stamp.png|link=Contact_the_BioCASe_team|200px|right|BioCASe Support]] ;[[Installation]]: Tutorial on how to install the BioCASe Provider Software. ;[[DatasourceSetup|Datasource Setup]]: How to set up a new BioCASe web service (aka data source). ;[[ABCD2Mapping|ABCD2 Mapping]]: Tutorial on mapping ABCD2. ;[[CommonABCD2Concepts|Common ABCD2 Concepts]]: List of mandatory, recommended and commonly used ABCD2 elements. ;[[SampleABCDDocument|Sample ABCD2 Document]]: Example ABCD2 document of a comprehensively mapped data provider. ;[[Debugging|Debugging Tutorial]]: Instructions on how to debug a BioCASe web service. ;[[Glossary|Glossary]]: List of terms and abbreviations used in this documentation. ;[[FAQ|FAQ]]: List of frequently asked questions and hopefully helpful answers. ;[[VersionHistory|Version History]]: Change list of current and past versions of the BioCASe Provider Software. 5325680dd98369461b4876c6aaa313e19692b166 225 215 2011-08-09T11:19:39Z JoergHoletschek 3 /* BioCASe Support */ wikitext text/x-wiki This is the Documentation Wiki of the BioCASe Provider Software. Here you will find tutorials on how to install, configure and debug the BioCASe Provider Software. More detailed information on BioCASe can be found in the [[BeginnersGuide|Beginner's Guide]] and on the [http://www.biocase.org/whats_biocase BioCASe homepage]. [[Image:BioCASeLogoLarge.jpg|link=http://www.biocase.org|150px|right|BioCASe]]Described in one paragraph, the BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. It is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. However, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the BioCASe network (http://www.biocase.org) and the Global Biodiversity Information Facility (http://www.gbif.org). After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. == BioCASe Support == [[Image:mail.png|link=Contact_the_BioCASe_team|150px|right|BioCASe Support]] Please get in contact with the BioCASe Support if you * have any questions concerning BioCASe in general or the installation/configuration of the Provider Software, * want to file a bug report, * have any comments, or * want to make a suggestion for a useful new feature in upcoming versions. You can either * use the ''Discussion'' tab of each page to leave us a comment, * use the from on the [[Contact_the_BioCASe_team|Contact page]] to send us a message, * or directly mail us to the address given on the [[Contact_the_BioCASe_team|Contact page]]. == Table of Contents == ;[[BeginnersGuide|Beginner's Guide]]: What the Provider Software is all about. ;[[Preparation]]: Things to be considered when publishing a database with BioCASe and potential preparatory steps needed. [[Image:support_stamp.png|link=Contact_the_BioCASe_team|200px|right|BioCASe Support]] ;[[Installation]]: Tutorial on how to install the BioCASe Provider Software. ;[[DatasourceSetup|Datasource Setup]]: How to set up a new BioCASe web service (aka data source). ;[[ABCD2Mapping|ABCD2 Mapping]]: Tutorial on mapping ABCD2. ;[[CommonABCD2Concepts|Common ABCD2 Concepts]]: List of mandatory, recommended and commonly used ABCD2 elements. ;[[SampleABCDDocument|Sample ABCD2 Document]]: Example ABCD2 document of a comprehensively mapped data provider. ;[[Debugging|Debugging Tutorial]]: Instructions on how to debug a BioCASe web service. ;[[Glossary|Glossary]]: List of terms and abbreviations used in this documentation. ;[[FAQ|FAQ]]: List of frequently asked questions and hopefully helpful answers. ;[[VersionHistory|Version History]]: Change list of current and past versions of the BioCASe Provider Software. 45be6b7306e592c464d059045072cf34d9e1447f 247 225 2011-08-09T12:06:29Z JoergHoletschek 3 /* BioCASe Support */ wikitext text/x-wiki This is the Documentation Wiki of the BioCASe Provider Software. Here you will find tutorials on how to install, configure and debug the BioCASe Provider Software. More detailed information on BioCASe can be found in the [[BeginnersGuide|Beginner's Guide]] and on the [http://www.biocase.org/whats_biocase BioCASe homepage]. [[Image:BioCASeLogoLarge.jpg|link=http://www.biocase.org|150px|right|BioCASe]]Described in one paragraph, the BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. It is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. However, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the BioCASe network (http://www.biocase.org) and the Global Biodiversity Information Facility (http://www.gbif.org). After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. == BioCASe Support == [[Image:mail.png|link=Special:Contact|150px|right|BioCASe Support]] Please get in contact with the BioCASe Support if you * have any questions concerning BioCASe in general or the installation/configuration of the Provider Software, * want to file a bug report, * have any comments, or * want to make a suggestion for a useful new feature in upcoming versions. You can either * use the ''Discussion'' tab of each page to leave us a comment, * use the from on the [[Special:Contact|Contact page]] to send us a message, * or directly mail us to the address given on the [[Special:Contact|Contact page]]. == Table of Contents == ;[[BeginnersGuide|Beginner's Guide]]: What the Provider Software is all about. ;[[Preparation]]: Things to be considered when publishing a database with BioCASe and potential preparatory steps needed. [[Image:support_stamp.png|link=Contact_the_BioCASe_team|200px|right|BioCASe Support]] ;[[Installation]]: Tutorial on how to install the BioCASe Provider Software. ;[[DatasourceSetup|Datasource Setup]]: How to set up a new BioCASe web service (aka data source). ;[[ABCD2Mapping|ABCD2 Mapping]]: Tutorial on mapping ABCD2. ;[[CommonABCD2Concepts|Common ABCD2 Concepts]]: List of mandatory, recommended and commonly used ABCD2 elements. ;[[SampleABCDDocument|Sample ABCD2 Document]]: Example ABCD2 document of a comprehensively mapped data provider. ;[[Debugging|Debugging Tutorial]]: Instructions on how to debug a BioCASe web service. ;[[Glossary|Glossary]]: List of terms and abbreviations used in this documentation. ;[[FAQ|FAQ]]: List of frequently asked questions and hopefully helpful answers. ;[[VersionHistory|Version History]]: Change list of current and past versions of the BioCASe Provider Software. 4c35608b87fd0d1d22f2947038f6550b88016bcb 261 247 2011-08-11T09:20:01Z JoergHoletschek 3 /* BioCASe Support */ wikitext text/x-wiki This is the Documentation Wiki of the BioCASe Provider Software. Here you will find tutorials on how to install, configure and debug the BioCASe Provider Software. More detailed information on BioCASe can be found in the [[BeginnersGuide|Beginner's Guide]] and on the [http://www.biocase.org/whats_biocase BioCASe homepage]. [[Image:BioCASeLogoLarge.jpg|link=http://www.biocase.org|150px|right|BioCASe]]Described in one paragraph, the BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. It is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. However, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the BioCASe network (http://www.biocase.org) and the Global Biodiversity Information Facility (http://www.gbif.org). After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. == BioCASe Support == [[Image:mail.png|link=Special:Contact|150px|right|BioCASe Support]] Please get in contact with the BioCASe Support if you * have any questions concerning BioCASe in general or the installation/configuration of the Provider Software, * want to file a bug report, * have any comments, or * want to make a suggestion for a useful new feature in upcoming versions. You can either * use the ''Discussion'' tab of each page to leave us a comment, * use the form on the [[Special:Contact|Contact page]] to send us a message, * or directly mail us to the address given on the [[Special:Contact|Contact page]]. == Table of Contents == ;[[BeginnersGuide|Beginner's Guide]]: What the Provider Software is all about. ;[[Preparation]]: Things to be considered when publishing a database with BioCASe and potential preparatory steps needed. [[Image:support_stamp.png|link=Contact_the_BioCASe_team|200px|right|BioCASe Support]] ;[[Installation]]: Tutorial on how to install the BioCASe Provider Software. ;[[DatasourceSetup|Datasource Setup]]: How to set up a new BioCASe web service (aka data source). ;[[ABCD2Mapping|ABCD2 Mapping]]: Tutorial on mapping ABCD2. ;[[CommonABCD2Concepts|Common ABCD2 Concepts]]: List of mandatory, recommended and commonly used ABCD2 elements. ;[[SampleABCDDocument|Sample ABCD2 Document]]: Example ABCD2 document of a comprehensively mapped data provider. ;[[Debugging|Debugging Tutorial]]: Instructions on how to debug a BioCASe web service. ;[[Glossary|Glossary]]: List of terms and abbreviations used in this documentation. ;[[FAQ|FAQ]]: List of frequently asked questions and hopefully helpful answers. ;[[VersionHistory|Version History]]: Change list of current and past versions of the BioCASe Provider Software. e1a9757c695b6969cb83c3a3806a37c740fd5f48 10 59 216 2011-08-09T10:46:39Z JoergHoletschek 3 Created page with "Please use this discussion page for * submitting comments to other users or to the developers of the Provider Software, * suggesting features for future versions of the Provider..." wikitext text/x-wiki Please use this discussion page for * submitting comments to other users or to the developers of the Provider Software, * suggesting features for future versions of the Provider Software * reporting bugs you've encountered, * any other hints you might have. Of course you can also [[Contact the BioCASe team]] directly. ---- 7a34a4bafcf9d56888ca63c037755fcc0ef8b139 217 216 2011-08-09T10:47:00Z JoergHoletschek 3 wikitext text/x-wiki Please use this discussion page for * submitting comments to other users or to the developers of the Provider Software, * suggesting features for future versions of the Provider Software, * reporting bugs you've encountered, * any other hints you might have. Of course you can also [[Contact the BioCASe team]] directly. ---- 1488280255444bcc454c4f55926db2b236096a31 218 217 2011-08-09T10:47:18Z JoergHoletschek 3 wikitext text/x-wiki Please use this discussion page for * submitting comments to other users or to the developers of the Provider Software, * suggesting features for future versions of the Provider Software, * reporting bugs you've encountered, * any other hints you might have. Of course you can also [[contact the BioCASe team]] directly. ---- 32b019601e4bd4f3909a15aefa6bc6e698aad8a3 220 218 2011-08-09T10:48:43Z JoergHoletschek 3 wikitext text/x-wiki Please use this discussion page for * submitting comments to other users or to the developers of the Provider Software, * suggesting features for future versions of the Provider Software, * reporting bugs you've encountered, * any other hints you might have. Of course you can also [[contact the BioCASe team]] directly. ae02e8a6641307bd1f17da63f9cf654c889f0773 233 220 2011-08-09T11:44:13Z JoergHoletschek 3 wikitext text/x-wiki Please use this discussion page for * submitting comments to other users or to the developers of the Provider Software, * suggesting features for future versions of the Provider Software, * reporting bugs you've encountered, * any other hints you might have. Of course you can also [[Special:Contact|contact the BioCASe team]] directly. c183c27a01e543f2e431c70efb166f49a8afedb3 257 233 2011-08-10T08:22:35Z JoergHoletschek 3 wikitext text/x-wiki Please use this discussion page for * submitting comments to other users or to the developers of the Provider Software, * suggesting features for future versions of the Provider Software, * reporting bugs you've encountered, * any other hints you might have. Click "Edit" on top of this page to leave your comment.<br>Of course you can also [[Special:Contact|contact the BioCASe team]] directly. e457d0956a17ff893d849cb38b67c2c365c0066b 1 60 219 2011-08-09T10:47:45Z JoergHoletschek 3 Created page with "{{Template:Discussion}}" wikitext text/x-wiki {{Template:Discussion}} 524dd381e0c2a8e7c25f04a08993e3cc71d7189a 221 219 2011-08-09T10:49:39Z JoergHoletschek 3 wikitext text/x-wiki {{Template:Discussion}} Please don't remove this template and add your comments below this line. ---- 888e4d32d434b800a78aec518600d01ec07cb2f3 222 221 2011-08-09T10:51:17Z JoergHoletschek 3 wikitext text/x-wiki {{Template:Discussion}} Please don't remove this template and add your comments below this line. ---- ;Short Description of your comment: Please write your comment here. 3e12c6a3c54978c5fadb90a134f15d6fc448252e 223 222 2011-08-09T11:01:48Z JoergHoletschek 3 wikitext text/x-wiki {{Template:Discussion}} <!--Please don't remove this template and add your comments below this line. --> -------------------------------------------------------------------------------- ;Short Description of your comment: Please write your comment here. 6caa82cedd1dce61e1ba7f1ac97ee2c4191c6847 224 223 2011-08-09T11:13:09Z JoergHoletschek 3 wikitext text/x-wiki {{Template:Discussion}} <!--Please add your comment below this line. --> -------------------------------------------------------------------------------- ;Short Description of your comment: Please write your comment here. 437b5ff4194e1283c0fc47af38be723a243b420e 231 224 2011-08-09T11:42:42Z 192.168.2.130 0 wikitext text/x-wiki {{Template:Discussion}} <!--Please add your comment below this line. --> -------------------------------------------------------------------------------- ;Short Description of your comment: Please write your comment here. test ee061cb45c7dd3f32e72c32a3be357afbdcff645 232 231 2011-08-09T11:43:10Z 192.168.2.130 0 wikitext text/x-wiki {{Template:Discussion}} <!--Please add your comment below this line. --> -------------------------------------------------------------------------------- ;Short Description of your comment: Please write your comment here. 437b5ff4194e1283c0fc47af38be723a243b420e 258 232 2011-08-10T08:36:09Z 192.168.2.130 0 wikitext text/x-wiki {{Template:Discussion}} <!--Please add your comment below this line. --> -------------------------------------------------------------------------------- ;Short Description of your comment: Please write your comment here. test ee061cb45c7dd3f32e72c32a3be357afbdcff645 259 258 2011-08-10T08:37:27Z JoergHoletschek 3 Reverted edits by [[Special:Contributions/192.168.2.130|192.168.2.130]] ([[User talk:192.168.2.130|talk]]) to last revision by [[User:JoergHoletschek|JoergHoletschek]] wikitext text/x-wiki {{Template:Discussion}} <!--Please add your comment below this line. --> -------------------------------------------------------------------------------- ;Short Description of your comment: Please write your comment here. 437b5ff4194e1283c0fc47af38be723a243b420e 8 61 227 2011-08-09T11:32:45Z JoergHoletschek 3 Created page with "If you have questions, comments, bug reports or suggestions for new features of the Provider Software, you can either * use the ''Discussion'' tab of each page to leave us a com..." wikitext text/x-wiki If you have questions, comments, bug reports or suggestions for new features of the Provider Software, you can either * use the ''Discussion'' tab of each page to leave us a comment, * use the from below to send us a message, * or directly mail us at [[File:support.png]]. In case you want to submit a bug report, please make sure to gather the following information when the error occurs: * your environment (operating system, web server, database management system), * the version of BioCASe you're running (can be found at the top of the BioCASe start page), * the steps that lead to the problem, * any error messages printed on screen (screenshots are a big plus), * any error or debug output that can be found in the logging files of the Provider Software (can be found in the ''log'' directory of your BioCASe installation) or in the log files of your web server. 0fc57b7681a6932bbfd9884c24a24e48d439e0e2 234 227 2011-08-09T11:48:02Z JoergHoletschek 3 wikitext text/x-wiki [[Image:mail.png|link=Contact_the_BioCASe_team|150px|right|BioCASe Support]]If you have questions, comments, bug reports or suggestions for new features of the Provider Software, you can either * use the ''Discussion'' tab of each page to leave us a comment, * use the from below to send us a message, * or directly mail us at [[File:support.png]]. In case you want to submit a bug report, please make sure to gather the following information when the error occurs: * your environment (operating system, web server, database management system), * the version of BioCASe you're running (can be found at the top of the BioCASe start page), * the steps that lead to the problem, * any error messages printed on screen (screenshots are a big plus), * any error or debug output that can be found in the logging files of the Provider Software (can be found in the ''log'' directory of your BioCASe installation) or in the log files of your web server. 54588b007990a524064fd55792d39b50c3b6a4f3 235 234 2011-08-09T11:48:25Z JoergHoletschek 3 wikitext text/x-wiki [[Image:mail.png|link=Contact_the_BioCASe_team|150px|right|BioCASe Support]]If you have questions, comments, bug reports or suggestions for new features of the Provider Software, you can either * use the ''Discussion'' tab of each page to leave us a comment, * use the from below to send us a message, * or directly mail us at [[File:support.png]]. '''In case you want to submit a bug report, please make sure to gather the following information when the error occurs:''' * your environment (operating system, web server, database management system), * the version of BioCASe you're running (can be found at the top of the BioCASe start page), * the steps that lead to the problem, * any error messages printed on screen (screenshots are a big plus), * any error or debug output that can be found in the logging files of the Provider Software (can be found in the ''log'' directory of your BioCASe installation) or in the log files of your web server. 713bf2302f23805a00c7df2840d8e2b126986cfc 8 3 228 203 2011-08-09T11:40:55Z JoergHoletschek 3 wikitext text/x-wiki * Table of Contents ** mainpage|mainpage-description ** BeginnersGuide|Beginner's Guide ** Preparation|Preparation ** Installation|Installation ** DatasourceSetup|Datasource Setup ** ABCD2Mapping|ABCD Mapping ** Debugging|Debugging ** Glossary|Glossary ** FAQ|FAQ * ABCD 2.06 ** CommonABCD2Concepts|Common Elements ** SampleABCDDocument|Sample Document * Feedback ** Comments|Comments Page ** [[Special:Contact]]|Contact us 8240e4307706b6aae01a83dda9d29d6fc52a3ac2 229 228 2011-08-09T11:41:16Z JoergHoletschek 3 wikitext text/x-wiki * Table of Contents ** mainpage|mainpage-description ** BeginnersGuide|Beginner's Guide ** Preparation|Preparation ** Installation|Installation ** DatasourceSetup|Datasource Setup ** ABCD2Mapping|ABCD Mapping ** Debugging|Debugging ** Glossary|Glossary ** FAQ|FAQ * ABCD 2.06 ** CommonABCD2Concepts|Common Elements ** SampleABCDDocument|Sample Document * Feedback ** Comments|Comments Page ** Special:Contact|Contact us 85342895985a61e2998b4e4904cd6f0db7fe4f21 230 229 2011-08-09T11:41:39Z JoergHoletschek 3 wikitext text/x-wiki * Table of Contents ** mainpage|mainpage-description ** BeginnersGuide|Beginner's Guide ** Preparation|Preparation ** Installation|Installation ** DatasourceSetup|Datasource Setup ** ABCD2Mapping|ABCD Mapping ** Debugging|Debugging ** Glossary|Glossary ** FAQ|FAQ * ABCD 2.06 ** CommonABCD2Concepts|Common Elements ** SampleABCDDocument|Sample Document * Feedback ** Special:Contact|Contact Us ceb03c75cf2e5ceff21edfc1c27c8aaca493697a 1 62 236 2011-08-09T11:55:21Z JoergHoletschek 3 Created page with "{{Template:Discussion}} <!--Please add your comment below this line. --> -------------------------------------------------------------------------------- ;Short Description of y..." wikitext text/x-wiki {{Template:Discussion}} <!--Please add your comment below this line. --> -------------------------------------------------------------------------------- ;Short Description of your comment: Please write your comment here. 437b5ff4194e1283c0fc47af38be723a243b420e 1 63 237 2011-08-09T11:55:45Z JoergHoletschek 3 Created page with "{{Template:Discussion}} <!--Please add your comment below this line. --> -------------------------------------------------------------------------------- ;Short Description of y..." wikitext text/x-wiki {{Template:Discussion}} <!--Please add your comment below this line. --> -------------------------------------------------------------------------------- ;Short Description of your comment: Please write your comment here. 437b5ff4194e1283c0fc47af38be723a243b420e 1 64 239 2011-08-09T11:56:28Z JoergHoletschek 3 Created page with "{{Template:Discussion}} <!--Please add your comment below this line. --> -------------------------------------------------------------------------------- ;Short Description of y..." wikitext text/x-wiki {{Template:Discussion}} <!--Please add your comment below this line. --> -------------------------------------------------------------------------------- ;Short Description of your comment: Please write your comment here. 437b5ff4194e1283c0fc47af38be723a243b420e 1 65 240 2011-08-09T11:56:39Z JoergHoletschek 3 Created page with "{{Template:Discussion}} <!--Please add your comment below this line. --> -------------------------------------------------------------------------------- ;Short Description of y..." wikitext text/x-wiki {{Template:Discussion}} <!--Please add your comment below this line. --> -------------------------------------------------------------------------------- ;Short Description of your comment: Please write your comment here. 437b5ff4194e1283c0fc47af38be723a243b420e 1 66 241 2011-08-09T11:56:53Z JoergHoletschek 3 Created page with "{{Template:Discussion}} <!--Please add your comment below this line. --> -------------------------------------------------------------------------------- ;Short Description of y..." wikitext text/x-wiki {{Template:Discussion}} <!--Please add your comment below this line. --> -------------------------------------------------------------------------------- ;Short Description of your comment: Please write your comment here. 437b5ff4194e1283c0fc47af38be723a243b420e 1 67 242 2011-08-09T11:57:01Z JoergHoletschek 3 Created page with "{{Template:Discussion}} <!--Please add your comment below this line. --> -------------------------------------------------------------------------------- ;Short Description of y..." wikitext text/x-wiki {{Template:Discussion}} <!--Please add your comment below this line. --> -------------------------------------------------------------------------------- ;Short Description of your comment: Please write your comment here. 437b5ff4194e1283c0fc47af38be723a243b420e 1 68 243 2011-08-09T11:57:14Z JoergHoletschek 3 Created page with "{{Template:Discussion}} <!--Please add your comment below this line. --> -------------------------------------------------------------------------------- ;Short Description of y..." wikitext text/x-wiki {{Template:Discussion}} <!--Please add your comment below this line. --> -------------------------------------------------------------------------------- ;Short Description of your comment: Please write your comment here. 437b5ff4194e1283c0fc47af38be723a243b420e 1 69 244 2011-08-09T11:57:22Z JoergHoletschek 3 Created page with "{{Template:Discussion}} <!--Please add your comment below this line. --> -------------------------------------------------------------------------------- ;Short Description of y..." wikitext text/x-wiki {{Template:Discussion}} <!--Please add your comment below this line. --> -------------------------------------------------------------------------------- ;Short Description of your comment: Please write your comment here. 437b5ff4194e1283c0fc47af38be723a243b420e 1 70 245 2011-08-09T11:57:31Z JoergHoletschek 3 Created page with "{{Template:Discussion}} <!--Please add your comment below this line. --> -------------------------------------------------------------------------------- ;Short Description of y..." wikitext text/x-wiki {{Template:Discussion}} <!--Please add your comment below this line. --> -------------------------------------------------------------------------------- ;Short Description of your comment: Please write your comment here. 437b5ff4194e1283c0fc47af38be723a243b420e 1 71 246 2011-08-09T11:58:25Z JoergHoletschek 3 Created page with "{{Template:Discussion}} <!--Please add your comment below this line. --> -------------------------------------------------------------------------------- ;Short Description of y..." wikitext text/x-wiki {{Template:Discussion}} <!--Please add your comment below this line. --> -------------------------------------------------------------------------------- ;Short Description of your comment: Please write your comment here. 437b5ff4194e1283c0fc47af38be723a243b420e 0 49 248 205 2011-08-09T12:07:12Z JoergHoletschek 3 /* Support */ wikitext text/x-wiki == What is the BioCASe Provider Software? == The BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. The Provider Software is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. The Provider Software is a suite of tools that need to be installed on the data provider’s web server. The core component is the PyWrapper, an XML/CGI database interface written in Python that allows a standardized access to a variety of database management systems and arbitrarily structured databases. Grouped around this are a number of tools for configuring, testing and debugging a BioCASe installation or web service. Even though BioCASe can be used for any conceptual XML data schema, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the [http://www.biocase.org BioCASe network] and the [http://www.gbif.org Global Biodiversity Information Facility]. == Requirements for Using the BPS == Before using the BioCASe Provider Software, two requirements need to be checked: * For one thing, the data to be published needs to be stored in an SQL compliant relational database management system (DBMS), with the data model being well documented or at least knowledge about the data model existing on someone’s head. Non-SQL DBMS are not supported. Currently, the BioCASe Provider Software can connect to the following databases: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * For installing BioCASe, you need to have a web server (Apache or Microsoft IIS) that is permanently connected to the Internet. This can be an institutional server of a hosting service (even though it’s usually a pain to get privileges to install software for the latter). Dial-up connections won’t work. == Steps == Once you’ve made sure the BioCASe Provider Software is the right choice for you ([[contact the BioCASe team]] if you feel unsafe with your decision), you need to do the following steps: # Think about which information you want to publish to the network. This involves settling the question of ownership of the data and the terms of use, license and usage restrictions for the published data. Also you should consider which information should '''not''' be published because they might affect endangered species (options are to block certain occurrences from publication or to blur the exact locality information). # Prepare the database for publication as described in the [[Preparation]] tutorial. This includes the decision on whether to publish the live database or a snapshot, the creation of a table for metadata and, in certain cases, some data transformation. # Follow the [[Installation]] Tutorial to install the BioCASe Provider Software. # Set up a BioCASe Datasource and connect it to your database, then configure the database structure as described in the [[DatasourceSetup]] tutorial. # Follow the [[ABCD2Mapping]] tutorial to create a mapping for the schema you want to use. # Test and debug your BioCASe web service. Read the [[Debugging]] tutorial to learn how to do this. # Once you’re done with setting up and configuring the BioCASe Provider Software, you can register the BioCASe web service with the information network you want to publish your data to, for example with [http://www.gbif.org GBIF]. We strongly recommend to [[contact the BioCASe team]] for checking your web service first as we might be able to foresee problems that are more difficult to resolve after registration. == Support == In case you run into difficulties during the installation and setup, you should * have a look at the [[FAQ]] which might list the solution for your question, * ask other BioCASe data providers for help, * get in contact with the [[Special:Contact|BioCASe team]]. Please describe exactly you environment (operating system, DBMS used), the steps that cause the problem and any error messages and hints you get. Screenshot are a big plus here. 1ee5e29b7f3d25e788ffde6e3ee47d48b269887e 251 248 2011-08-09T12:15:00Z JoergHoletschek 3 /* Steps */ wikitext text/x-wiki == What is the BioCASe Provider Software? == The BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. The Provider Software is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. The Provider Software is a suite of tools that need to be installed on the data provider’s web server. The core component is the PyWrapper, an XML/CGI database interface written in Python that allows a standardized access to a variety of database management systems and arbitrarily structured databases. Grouped around this are a number of tools for configuring, testing and debugging a BioCASe installation or web service. Even though BioCASe can be used for any conceptual XML data schema, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the [http://www.biocase.org BioCASe network] and the [http://www.gbif.org Global Biodiversity Information Facility]. == Requirements for Using the BPS == Before using the BioCASe Provider Software, two requirements need to be checked: * For one thing, the data to be published needs to be stored in an SQL compliant relational database management system (DBMS), with the data model being well documented or at least knowledge about the data model existing on someone’s head. Non-SQL DBMS are not supported. Currently, the BioCASe Provider Software can connect to the following databases: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * For installing BioCASe, you need to have a web server (Apache or Microsoft IIS) that is permanently connected to the Internet. This can be an institutional server of a hosting service (even though it’s usually a pain to get privileges to install software for the latter). Dial-up connections won’t work. == Steps == Once you’ve made sure the BioCASe Provider Software is the right choice for you ([[Special:Contact|contact the BioCASe team]] if you feel unsafe with your decision), you need to do the following steps: # Think about which information you want to publish to the network. This involves settling the question of ownership of the data and the terms of use, license and usage restrictions for the published data. Also you should consider which information should '''not''' be published because they might affect endangered species (options are to block certain occurrences from publication or to blur the exact locality information). # Prepare the database for publication as described in the [[Preparation]] tutorial. This includes the decision on whether to publish the live database or a snapshot, the creation of a table for metadata and, in certain cases, some data transformation. # Follow the [[Installation]] Tutorial to install the BioCASe Provider Software. # Set up a BioCASe Datasource and connect it to your database, then configure the database structure as described in the [[DatasourceSetup]] tutorial. # Follow the [[ABCD2Mapping]] tutorial to create a mapping for the schema you want to use. # Test and debug your BioCASe web service. Read the [[Debugging]] tutorial to learn how to do this. # Once you’re done with setting up and configuring the BioCASe Provider Software, you can register the BioCASe web service with the information network you want to publish your data to, for example with [http://www.gbif.org GBIF]. We strongly recommend to [[Special:Contact|contact the BioCASe team]] for checking your web service first as we might be able to foresee problems that are more difficult to resolve after registration. == Support == In case you run into difficulties during the installation and setup, you should * have a look at the [[FAQ]] which might list the solution for your question, * ask other BioCASe data providers for help, * get in contact with the [[Special:Contact|BioCASe team]]. Please describe exactly you environment (operating system, DBMS used), the steps that cause the problem and any error messages and hints you get. Screenshot are a big plus here. c96922d3423a0dcb14de44b7584c70760b1e2ffb 0 17 249 171 2011-08-09T12:07:47Z JoergHoletschek 3 /* Repeatable Elements in ABCD */ wikitext text/x-wiki Before you start publishing your database with BioCASe, you need to consider several issues. In some cases, some preparation of your database for the publication might be necessary. You should read this page of the wiki careful before you start configuring the Provider Software for a given database. == Live-DB versus Snapshot == The easiest way is to connect BioCASe to the same database you’re working with. This would result in any changes you do being visible on the BioCASe access point right away – which is an idea you might not like. Some curators prefer to publish defined versions of their database, for example after they’ve finished reviewing all specimens of a certain genus. If you prefer this, you need to create a second database that holds defined snapshots of your working database and that will be published instead. Snapshots could be generated in a regular interval (e.g., each Friday evening) or upon request. Please contact your IT guys for more help on this. == Access Control == The BioCASe Provider Software will only publish your data to biodiversity networks, meaning it will only read from the database and never do any changes. For security reasons it’s a good idea to create a user for BioCASe that has only read privileges on the database – more precisely on the tables you want to publish. Since the configuration tool will try to load a list of tables and columns from the database to allow easier setup, this user should also be allowed to load meta-information (<small>SHOW TABLES, SHOW COLUMNS</small>). == Metadata == Metadata describe your dataset as a whole - title, description, curator’s name, ownership, date of last modification, terms of use etc. BioCASe can publish a lot of metadata – and you should take the time to provide as much information as you can, since this will increase the usability of your published data. For storing metadata, please create an extra table in your database and fill it in. If you go to the BioCASe folder of your installation, you’ll find the template for this table in ''sampledb/abcdmetadata''. Either use the SQL script to create it in your database, or import it from the Access database (''metadata.mdb''). Per default, all records of your database will make up one dataset, being described by one set of metadata. In case your database holds records of several datasets, you can also group these into several datasets in ABCD by providing different sets of metadata. So when you realize during gathering of meta-information that they’re not the same for all records, this is an indicator that your database must be grouped into several datasets. If all records to be published share the same metadata, the metadata table needs to have just one row. If they don’t, it needs to hold several rows, one set of metadata for each dataset, identified by a unique primary key (1, 2, …). In order to associate each record of your database with the correct set of metadata, you will need to add a foreign key into the base record table that links to the respective entry in the metadata table. Ask your IT guys if you need help on that. == Controlled Denormalisation == As you will see during configuration, the Provider Software is able to join together the different tables of your database’s data model in order to retrieve all information to be published from the database. This works good if the number of tables is within reasonable limits, that is up to approximately 10-15. Data models used by off-the-shelf collection management system are usually highly normalised and use far more tables. Specify’s data model, for example, has about 70 tables. Configuring such a complex model in BioCASe will be a terrible pain; moreover, in case your database holds a huge number of records, the time required by the Provider Software for answering search request will become too long. Therefore it is advisable in such cases to add some controlled denormalisation to your database, that is to export the data to be published into several cache tables. These tables could be stored in the same database as the source tables or separate in an extra cache database, maybe even on a different server. They could be created in a regular interval (e.g. each night) or upon request by SQL scripts. The figure below shows this denormalisation process for the data model of Specify. On the left side, you see some of Specify’s source tables (only the ones starting with the letters A, B, C and D). With an SQL script they will be reduced to 7 cache tables, one of them being abcd_object. These tables hold only the columns that will be published, drawn from up to 15 source tables. [[image:prepDenormalisation.png|750px]] == Repeatable Elements in ABCD == One of the main features of ABCD (the schema you will probably use to publish your data) is repeatable elements. For example, ABCD can store multiple identifications or several URLs for multimedia objects attached to a specimen record. All these URLs will be stored in the same ABCD element – namely ''MultimediaObject/FileURI''. Another example is the higher taxonomy (all taxa above genus level). Typically these are stored in different columns in databases (see figure below). In ABCD, they will all end up in the same element, ''HigherTaxon/Name''. In order to be able to map the different ranks stored in columns to the same ABCD concept, we need to pivot these columns into rows – each rank kept in a column needs to be turned into a row, with two columns storing taxon name and rank, one column for linking to the according specimen record, plus one newly created primary key: [[File:prepPivoting.png]] Such a transformation can be easily done with a database view. Views do exactly what the name suggests – they provide a different view on the data. They can be used like regular tables, but they merely represent a different representation of another table. Updates of the underlying tables (or insert/deletions) will also affect views created on these tables, since the data is stored only once in the database. Let’s assume you have the higher taxonomy stored in the specimen table like this: [[File:prepEditTable.png]] The SQL statement <pre> CREATE VIEW [dbo].[vwHigherTaxa] AS SELECT 'k_' + [EDIT_ATBI_RecordID] AS id, [EDIT_ATBI_RecordID] AS unit_id, [kingdom] AS name, 'kingdom' AS rank FROM unit_data WHERE [kingdom] IS NOT NULL UNION ALL SELECT 'p_' + [EDIT_ATBI_RecordID], [EDIT_ATBI_RecordID], [phylum], 'phylum'‚ FROM unit_data WHERE [phylum] IS NOT NULL UNION ALL ... </pre> will create a view that meets the requirements described above. Remember that even though it looks like a table, it’s just another representation of the data stored in the table unit_data: [[File:prepEditView.png]] Such pivoting must be done in all cases when several columns need to be mapped to the same (repeatable) ABCD element. If it’s just one value to be mapped, there is no need for that. Commonly used ABCD elements that are repeatable and might ask for such preparatory transformations are * Identification * HigherTaxon * GatheringSite/NamedArea * Metadata/Scope/GeoecologicalTerms * Metadata/Scope/TaxonomicTerms * MultimediaObjects * MeasurementsOrFacts If you have problems preparing your database for publication, please [[Special:Contact|contact the BioCASe team]]. e958d081a094462f73a15fa76a631e2eeb9a390e 0 6 250 210 2011-08-09T12:08:23Z JoergHoletschek 3 /* Requirements */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in (in case Python is not found, you’ll have to add the full path name to the Python executable):<pre>python setup.py</pre>Make sure the Python version given at the beginning out the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] All of the questions do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Import configuration from existing installation:</strong> Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.foobar.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.foobar.org/biocase'', for example. After answering these three questions, the script will update your files. At the end of the output it will print the lines you will need later to create the link, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so will compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.foobar.org/biocase''. If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can a pain; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the DBMS used by the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to he library test page that shows the status of installed libraries. On a vanilla BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries ship with the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' in the browser window), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown [[Installation#Linking_BioCASe_and_Your_Web_Server|here]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] 32d59c1f1298539ffb5f9da94bac6a73d58b30ad 0 41 252 136 2011-08-09T12:17:54Z JoergHoletschek 3 wikitext text/x-wiki This tutorial explains how to debug a BioCASe web service and a BioCASe installation in general. If you follow the instructions and don’t manage to get your service running, please [[Special:Contact|contact the BioCASe team]]. == Debugging a BioCASe Web Service == Debugging a BioCASe web service can be done with the manual Query Form. It allows you to load a BioCASe request (Search/Scan) from a template, edit the parameters of the request (e.g. the filter) and will show you the response document, which holds the requested ABCD records and useful debug information. === Opening the Query Form === There are several ways to open the Query Form. On the entry page of a datasource, <small>TEST and DEBUG</small> will open the Query Form: [[File:debugDatasourceEntryPage.png]] In the data source configuration tool, the link to the according query form can be always found in the link list at the top of the page. Moreover, in the mapping editor, the button <small>Test mapping!</small> will open the Query Form in a separate tab: [[File:abcdMappingEditorEmptyTop.png]] === Basic Usage === On top of the Query Form you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ''flora'' being the name of the datasource. Make sure this is the correct access point of the web service you want to debug! [[File:debugQueryformFull.png]] The usage of the Query Form is very straightforward: * Click on one of the links below the text box to load a template, for example for an ABCD2 Scan or ABCD2 Search; * Edit the request parameters; * Click on <small>Submit</small> to get the response document. The section <small>BioCASe protocol filter operators</small> lists the operators that can be used for creating complex filters that combine several filter criteria. More about creating filters can be found in the BioCASe Protocol documentation at http://www.biocase.org/products/protocols/index.shtml. Clicking on the <small>Show</small> link in the <small>Capabilities</small> section will show the Capabilities response in a frame of the Query Form. This is useful, since it shows the namespaces for the schemas supported by the web service (which is required in the request text) and XPaths of mapped ABCD elements (required for creating filters on these elements). By using Copy & Paste you can spare the trouble of entering these monstrous strings of characters. === Changing the Debug Level for a Data Source === Debug info can be set to one of four levels for a BioCASe data source: * <strong>Error</strong> is the highest level will only output errors that prevent the web server from answering a request. * <strong>Warning</strong> will output errors that keep BioCASe from answering a request and warnings that don’t prevent the fulfilling of the request, but should be analysed carefully. For example, they might tell you that records have been dropped because mandatory ABCD elements have not been mapped or were empty for certain records. * <strong>Info</strong> is the default and will add some more information about how the Provider Software processes the request. * <strong>Debug</strong> is the most detailed level and will produce very verbose debug output. It adds very comprehensive information about how the ABCD documents are constructed and is suitable only for experienced users. For most cases, keeping the default ''Info'' level should be sufficient for testing and debugging a web service. For production use, you should set it to ''Error'' later. You can do this in the configuration of a datasource under Settings: [[File:debugDatasourceSettings.png]] == Analyzing the Response Document == Upon clicking <small>Submit</small>, the Query Form will send the request to the web service and wait for the response. It might take some time – up to minutes – until the response document shows up. Amongst others, this depends on * the complexity of the query (especially the filters), * the size of the dataset returned, * the debug level set (“debug” results in very exhaustive debug output), * the performance of the servers running BioCASe and the database, * the network bandwidth. The response document consists of three sections (a full response document can be found [[CommonABCD2Concepts|here]]): [[File:debugResponseDocumentOverview.png]] === Header === This section contains information about the BioCASe installation – operating system, DBMS used, Python/BioCASe version etc. It is intended for a consumer application and won’t be very helpful for debugging: <pre> <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> </pre> === Content === This section makes up the payload of the response document, depending on the type of request: <strong>Capabilities:</strong> Lists all schemas supported by the web service. For each schema, all mapped concepts (identified by their XPaths) are listed with data type and searchable flag. <strong>Scan:</strong> The document will simply list all distinct values for the concept specified in the request. When done for the scientific name, it would look similar to this: <pre> <biocase:content recordCount="1664" recordDropped="0" recordStart="0" totalSearchHits="1664"> <biocase:scan> <biocase:value>Achillea atrata L.</biocase:value> <biocase:value>Achillea millefolium L.</biocase:value> <biocase:value>Achillea nobilis L. ssp. nobilis</biocase:value> <biocase:value>Achillea ptarmica L.</biocase:value> <biocase:value>Acinos alpinus (L.) Moench</biocase:value> <biocase:value>Acinos arvensis (Lam.) Dandy</biocase:value> <biocase:value>Aconitum degenii ssp. paniculatum (Arcang.) Mucher</biocase:value> <biocase:value>Aconitum lycoctonum L. ssp. lycoctonum</biocase:value> <biocase:value>Aconitum napellus L. s.l.</biocase:value> ... </biocase:scan> </biocase:content> </pre> <strong>Search:</strong> For this type of request, the content consists of the ABCD dataset(s). See the the [[CommonABCD2Concepts| ABCD2 example]]. === Diagnostics (Debug Output) === This section is the most interesting part for debugging. If debug level is set to ''Info'' (which is recommended for debugging), it will document the processing of the request by the web service. The output is easy to understand, for example it will contain the SQL statements sent to the database. For a successful search request it could look similar to this: <pre> <biocase:diagnostics> <biocase:diagnostic severity="INFO"> Datasource wrapper FloraExsiccataBavarica requested </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Reading PSF from C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica \provider_setup_file.xml </biocase:diagnostic> <biocase:diagnostic severity="DEBUG"> PSF: PSF=C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\provider_setup_file.xml, recLimit=100, loglevel=20, user=webuser, database=rbg, dbIP=192.168.2.10, dbms=mysql, encoding=latin_1, schemas={u'http://www.tdwg.org/schemas/abcd/2.06': <biocase.wrapper.psf_handler.SupportedSchema instance at 0x011F5B98>}, tablegraph=GRAPH: graph: unit-metadata, +++ ALIAS2TABLE: {u'unit': u'feb3', u'metadata': u'metadata'} </biocase:diagnostic> <biocase:diagnostic severity="INFO"> BioCASe protocol used. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> HTTP parameter 'query' used for building the request. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT DISTINCT unit.id FROM feb3 AS unit WHERE (unit.nameautor LIKE 'Allium %' ) LIMIT 11' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT unit.id, metadata.supplier_organisation, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (40, 41, 42, 43, 44, 45, 46, 47)) ORDER BY unit.id' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> time to execute request is 0.46599984169 </biocase:diagnostic> </biocase:diagnostics> </pre> == Sample Errors and Warnings in the Debug Output == <pre> <biocase:diagnostic severity="ERROR"> No DB connection could be established. Please verify DB state and connection parameters in your provider_setup_file.xml. </biocase:diagnostic> </pre> Well... just what it says. The Provider Software could not connect to the database to be published. Check the database connection settings (server IP, database name, user, and password) and make sure you’ve installed the Python package for the DBMS you’re using (see the <small>Libs test</small> page in <small>Utilities</small>). <pre> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\flora\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="ERROR"> The CMF DB mapping file http://www.tdwg.org/schemas/abcd/2.06 was corrupt or could not be interpreted. </biocase:diagnostic> </pre> The requested schema (identified by the namespace http://www.tdwg.org/schemas/abcd/2.06) is not supported by that service. Create a mapping and map elements. <pre> <biocase:diagnostic severity="ERROR"> The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result /TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts </biocase:diagnostic> </pre> Obvious - happens if you try to use a concept that is not mapped in a scan request or a filter. <pre> <biocase:diagnostic severity="ERROR"> A SQL statement produced an error: SELECT unit.id, metadata.supplier_organisation, metadata.supplier_email, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) ORDER BY unit.id </pre> That could have a number of reasons: One of the table/column names that were set up in the configuration does not exist (because it was renamed or removed) or the credentials used by the BPS do not have sufficient privileges. Simply copy the SQL statement and execute it manually on the database with a regular database client that will show you the detailed error message returned by the DBMS. <pre> <biocase:diagnostic severity="WARNING"> The mandatory child element SourceInstitutionID could not be created [0 existing, but 1 required]. The element /DataSets/DataSet/Units/Unit was also dropped. </biocase:diagnostic> <biocase:diagnostic severity="WARNING"> The mandatory child element Units could not be created [0 existing, but 1 required]. The element /DataSets/DataSet was also dropped. </biocase:diagnostic> </pre> One problem caused the next one in this sequence of warnings: The mandatory ABCD element ''SourceInstitutionID '' was either not mapped or empty; therefore the respective record (''unit'') was dropped. Because this happened for all units in the dataset, all units were dropped, resulting in an empty ''Units'' sub tree. The second warning tells you what happened then: Because the ''Units'' element is mandatory for the ''Dataset'' element, the whole dataset was dropped. As a result, the response ABCD document was empty. == Debugging a BioCASe Installation == In case the debug output of the web service doesn’t help you to find the problem, or if the BPS behaves not as intended without displaying any error messages, you should have a look at the debug logs written by the BPS. To turn on the Debug logs, go to the System Administration page and set <small>Debug logs</small> to <small>True</small>. [[File:instConfigTool.png|500px]] Redo the step that didn’t work and open the ''log'' folder of your BioCASe installation. There are several global log files (pywrapper_request.log, pywrapper_error.log, webapp_debug.log, webapp_error.log) and a debug files per data source (e.g. debug_flora.log). Open these files and see if they can give you a clue to the problem. If you still can’t get it running, [[Special:Contact|contact the BioCASe team]]. cd913e69b96767c7121d27b995620ee02ec78c10 0 52 253 209 2011-08-09T12:18:34Z JoergHoletschek 3 /* I get a nice pink “A problem occurred in a Python script” error. What’s wrong? */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===When can I use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just contact us and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I’ve forgot the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might just found a bag. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[contact the BioCASe team|contact us]]; we’ll try our best to help you. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the [[Comments]] page or get directly in [[contact the BioCASe team|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry (you might need to close and reopen your browser). If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I cannot find the installation package for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. 5330edeb878a1beb398b8cd7b153d78ff80fa95d 254 253 2011-08-09T12:18:51Z JoergHoletschek 3 /* I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me. */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===When can I use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just contact us and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I’ve forgot the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might just found a bag. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the [[Comments]] page or get directly in [[contact the BioCASe team|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry (you might need to close and reopen your browser). If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I cannot find the installation package for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. 838872b58507bbde449a033f3ebcc4e1806a10f9 255 254 2011-08-09T12:19:52Z JoergHoletschek 3 /* I have some recommendations/bug report. How can I file it? */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===When can I use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just contact us and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I’ve forgot the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might just found a bag. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry (you might need to close and reopen your browser). If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I cannot find the installation package for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. 0919cde49fc3345ef2b2106d9e956c88c8fc1965 8 72 256 2011-08-09T15:09:12Z JoergHoletschek 3 Created page with "/* CSS placed here will affect the print output */ #BreadCrumbsTrail { display:none; } #firstHeading { margin-top: 20px; }" css text/css /* CSS placed here will affect the print output */ #BreadCrumbsTrail { display:none; } #firstHeading { margin-top: 20px; } 9d3dc61ce3e17979693be282fd9163be4d824164 0 23 260 208 2011-08-11T08:41:17Z JoergHoletschek 3 /* Version 2.6.1 [upcoming] */ wikitext text/x-wiki == Version 2 History == ===Version 2.6.1 [upcoming]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.5.5 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. See example [http://ww3.bgbm.org/bps2/ScreenShots?action=AttachFile&do=get&target=config_tool4.gif here] * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 4858eefdef2c4ed78790c4ddc7b83ed198034228 8 61 262 235 2011-08-11T09:21:47Z JoergHoletschek 3 wikitext text/x-wiki [[Image:mail.png|150px|right|BioCASe Support]]If you have questions, comments, bug reports or suggestions for new features of the Provider Software, you can either * use the ''Discussion'' tab of each page to leave us a comment, * use the from below to send us a message, * or directly mail us at [[File:support.png]]. '''In case you want to submit a bug report, please make sure to gather the following information when the error occurs:''' * your environment (operating system, web server, database management system), * the version of BioCASe you're running (can be found at the top of the BioCASe start page), * the steps that lead to the problem, * any error messages printed on screen (screenshots are a big plus), * any error or debug output that can be found in the logging files of the Provider Software (can be found in the ''log'' directory of your BioCASe installation) or in the log files of your web server. ae2166a98404fa61a09f45d273f1b42613cf7e55 264 262 2011-08-11T09:26:10Z JoergHoletschek 3 wikitext text/x-wiki [[Image:mail.png|150px|right|BioCASe Support]]If you have questions, comments, bug reports or suggestions for new features of the Provider Software, you can either * use the ''Discussion'' tab of each page to leave us a comment, * use the form below to send us a message, * or mail us directly at &nbsp;[[File:support.png]]. '''In case you want to submit a bug report, please make sure to gather the following information when the error occurs:''' * your environment (operating system, web server, database management system), * the version of BioCASe you're running (can be found at the top of the BioCASe start page), * the steps that caused the problem, * any error messages printed on screen (screenshots are a big plus), * any error or debug output that can be found in the logging files of the Provider Software (can be found in the ''log'' directory of your BioCASe installation) or in the log files of your web server. 80427c985766970ad57753e330c04e8468a05eba 0 1 263 261 2011-08-11T09:23:39Z JoergHoletschek 3 /* Table of Contents */ wikitext text/x-wiki This is the Documentation Wiki of the BioCASe Provider Software. Here you will find tutorials on how to install, configure and debug the BioCASe Provider Software. More detailed information on BioCASe can be found in the [[BeginnersGuide|Beginner's Guide]] and on the [http://www.biocase.org/whats_biocase BioCASe homepage]. [[Image:BioCASeLogoLarge.jpg|link=http://www.biocase.org|150px|right|BioCASe]]Described in one paragraph, the BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. It is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. However, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the BioCASe network (http://www.biocase.org) and the Global Biodiversity Information Facility (http://www.gbif.org). After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. == BioCASe Support == [[Image:mail.png|link=Special:Contact|150px|right|BioCASe Support]] Please get in contact with the BioCASe Support if you * have any questions concerning BioCASe in general or the installation/configuration of the Provider Software, * want to file a bug report, * have any comments, or * want to make a suggestion for a useful new feature in upcoming versions. You can either * use the ''Discussion'' tab of each page to leave us a comment, * use the form on the [[Special:Contact|Contact page]] to send us a message, * or directly mail us to the address given on the [[Special:Contact|Contact page]]. == Table of Contents == ;[[BeginnersGuide|Beginner's Guide]]: What the Provider Software is all about. ;[[Preparation]]: Things to be considered when publishing a database with BioCASe and potential preparatory steps needed. [[Image:support_stamp.png|link=Special:Contact|200px|right|BioCASe Support]] ;[[Installation]]: Tutorial on how to install the BioCASe Provider Software. ;[[DatasourceSetup|Datasource Setup]]: How to set up a new BioCASe web service (aka data source). ;[[ABCD2Mapping|ABCD2 Mapping]]: Tutorial on mapping ABCD2. ;[[CommonABCD2Concepts|Common ABCD2 Concepts]]: List of mandatory, recommended and commonly used ABCD2 elements. ;[[SampleABCDDocument|Sample ABCD2 Document]]: Example ABCD2 document of a comprehensively mapped data provider. ;[[Debugging|Debugging Tutorial]]: Instructions on how to debug a BioCASe web service. ;[[Glossary|Glossary]]: List of terms and abbreviations used in this documentation. ;[[FAQ|FAQ]]: List of frequently asked questions and hopefully helpful answers. ;[[VersionHistory|Version History]]: Change list of current and past versions of the BioCASe Provider Software. 707e947fd05c6995b39f7768c7e7443d7a8faff3 8 2 265 214 2011-08-11T10:15:35Z JoergHoletschek 3 css text/css /* CSS placed here will be applied to all skins */ /* Hide title of Main_Page */ /* body.page-Main_Page h1.firstHeading { display:none; } */ /* Hide every title except special pages */ /*body.ns-subject h1.firstHeading { display:none; }*/ /* Added manually: Hide toolbox */ #p-tb{ display:none;} /* Added manually: Move BreadcrumbsTrail below Heading */ /*#BreadCrumbsTrail { margin-bottom: -30px; position: relative; top: -70px; }*/ #BreadCrumbsTrail { margin-bottom: -30px; position: absolute; top: 5px; } #firstHeading { margin-top: 20px; } 82694d05e72693e5ddbe1ae20aa57713917932fb 266 265 2011-08-11T10:17:21Z JoergHoletschek 3 css text/css /* CSS placed here will be applied to all skins */ /* Hide title of Main_Page */ /* body.page-Main_Page h1.firstHeading { display:none; } */ /* Hide every title except special pages */ /*body.ns-subject h1.firstHeading { display:none; }*/ /* Added manually: Hide toolbox */ #p-tb{ display:none;} /* Added manually: Move BreadcrumbsTrail below Heading */ #BreadCrumbsTrail { margin-bottom: -30px; position: relative; top: -70px; } /*#BreadCrumbsTrail { margin-bottom: -30px; position: absolute; top: 5px; }*/ #firstHeading { margin-top: 20px; } 154772a1d461a7a43c5ec5d626dcbb18c2566d48 270 266 2011-08-11T12:36:17Z JoergHoletschek 3 css text/css /* CSS placed here will be applied to all skins */ /* Hide title of Main_Page */ /* body.page-Main_Page h1.firstHeading { display:none; } */ /* Hide every title except special pages */ /*body.ns-subject h1.firstHeading { display:none; }*/ /* Added manually: Hide toolbox */ #p-tb{ display:none;} /* Added manually: Move BreadcrumbsTrail below Heading */ #BreadCrumbsTrail { margin-bottom: -30px; position: relative; top: -70px; } #firstHeading { margin-top: 20px; } div.mw-geshi { /*damit die Codebox so aussieht, wie im Monobook*/ padding: 1em; margin: 1em 0; border: 1px dashed #2f6fab; /*Scrollbalken, falls der Text zu breit ist# overflow:auto; damit der Code unabhängig von der Skin gleich groß angezeigt wird font-size: 10pt;*/ } 1391e73e9b04877972e4850709eac5cc88416cb3 271 270 2011-08-11T12:37:48Z JoergHoletschek 3 css text/css /* CSS placed here will be applied to all skins */ /* Hide title of Main_Page */ /* body.page-Main_Page h1.firstHeading { display:none; } */ /* Hide every title except special pages */ /*body.ns-subject h1.firstHeading { display:none; }*/ /* Added manually: Hide toolbox */ #p-tb{ display:none;} /* Added manually: Move BreadcrumbsTrail below Heading */ #BreadCrumbsTrail { margin-bottom: -30px; position: relative; top: -70px; } #firstHeading { margin-top: 20px; } div.mw-geshi { /*damit die Codebox so aussieht, wie im Monobook*/ padding: 1em; margin: 1em 0; border: 1px dashed #2f6fab; /*Scrollbalken, falls der Text zu breit ist*/ # overflow:auto; /*damit der Code unabhängig von der Skin gleich groß angezeigt wird font-size: 10pt;*/ } 4b57f5d5afc4e9ec039ec5a97dfd7c6469dec3f9 272 271 2011-08-11T12:38:44Z JoergHoletschek 3 css text/css /* CSS placed here will be applied to all skins */ /* Hide title of Main_Page */ /* body.page-Main_Page h1.firstHeading { display:none; } */ /* Hide every title except special pages */ /*body.ns-subject h1.firstHeading { display:none; }*/ /* Added manually: Hide toolbox */ #p-tb{ display:none;} /* Added manually: Move BreadcrumbsTrail below Heading */ #BreadCrumbsTrail { margin-bottom: -30px; position: relative; top: -70px; } #firstHeading { margin-top: 20px; } div.mw-geshi { /*damit die Codebox so aussieht, wie im Monobook*/ padding: 1em; margin: 1em 0; border: 1px dashed #2f6fab; /*Scrollbalken, falls der Text zu breit ist*/ #overflow:auto; /*damit der Code unabhängig von der Skin gleich groß angezeigt wird font-size: 10pt;*/ } 32d9f01221d73ecfc3fc161d8b751622a685cbb5 273 272 2011-08-11T12:39:47Z JoergHoletschek 3 css text/css /* CSS placed here will be applied to all skins */ /* Hide title of Main_Page */ /* body.page-Main_Page h1.firstHeading { display:none; } */ /* Hide every title except special pages */ /*body.ns-subject h1.firstHeading { display:none; }*/ /* Added manually: Hide toolbox */ #p-tb{ display:none;} /* Added manually: Move BreadcrumbsTrail below Heading */ #BreadCrumbsTrail { margin-bottom: -30px; position: relative; top: -70px; } #firstHeading { margin-top: 20px; } /* Codeboxen */ div.mw-geshi { padding: 1em; margin: 1em 0; border: 1px dashed #2f6fab; #overflow:auto; } 854eceeee3d04779a14a81b07909928ebdbc7292 274 273 2011-08-11T12:41:06Z JoergHoletschek 3 css text/css /* CSS placed here will be applied to all skins */ /* Hide title of Main_Page */ /* body.page-Main_Page h1.firstHeading { display:none; } */ /* Hide every title except special pages */ /*body.ns-subject h1.firstHeading { display:none; }*/ /* Added manually: Hide toolbox */ #p-tb{ display:none;} /* Added manually: Move BreadcrumbsTrail below Heading */ #BreadCrumbsTrail { margin-bottom: -30px; position: relative; top: -70px; } #firstHeading { margin-top: 20px; } /* Codeboxen */ div.mw-geshi { padding: 1em; margin: 1em 0; border: 1px dashed #2f6fab; overflow:auto; } 912d8ec44ddc7533162db7c9b59fc4e884c84cab 275 274 2011-08-11T12:42:12Z JoergHoletschek 3 css text/css /* CSS placed here will be applied to all skins */ /* Hide title of Main_Page */ /* body.page-Main_Page h1.firstHeading { display:none; } */ /* Hide every title except special pages */ /*body.ns-subject h1.firstHeading { display:none; }*/ /* Added manually: Hide toolbox */ #p-tb{ display:none;} /* Added manually: Move BreadcrumbsTrail below Heading */ #BreadCrumbsTrail { margin-bottom: -30px; position: relative; top: -70px; } #firstHeading { margin-top: 20px; } /* Codeboxen */ div.mw-geshi { padding: 1em; margin: 1em 0; border: 1px dashed #2f6fab; overflow:auto; font-size: 10pt; } 19432b3383ee1e57979a1d5120b249cfaf7f1e83 0 52 267 255 2011-08-11T11:38:53Z JoergHoletschek 3 /* I get a nice pink “A problem occurred in a Python script” error. What’s wrong? */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===When can I use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just contact us and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I’ve forgot the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry (you might need to close and reopen your browser). If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I cannot find the installation package for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. 54653ad99bfac566da2383ca80f56e9ba7d79bb0 291 267 2011-08-15T11:57:29Z JoergHoletschek 3 wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===When can I use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just contact us and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I’ve forgot the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry (you might need to close and reopen your browser). If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I cannot find the installation package for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. == Trouble Shooting == 29ee2616640b8b01f5e16720d0a6aaf5489690bd 292 291 2011-08-15T11:58:38Z JoergHoletschek 3 wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===When can I use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just contact us and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry (you might need to close and reopen your browser). If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I cannot find the installation package for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. == Trouble Shooting == ===I’ve forgot the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. ccac12920803665cf0ae80d3df69d6b90022db94 293 292 2011-08-15T12:16:47Z JoergHoletschek 3 /* Trouble Shooting */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===When can I use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just contact us and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry (you might need to close and reopen your browser). If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I cannot find the installation package for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. == Trouble Shooting == ===I’ve forgot the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that tells me a Python settings file (sitecustomize.py) will be changed. Unfortunately, on my installation there is no such file in that location; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map it to the database columns. If, after thinking it over again, you still want to map literals, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. b091fdbe2cf78ed6828c3f9cebfdb9e88993f5dd 294 293 2011-08-15T12:17:27Z JoergHoletschek 3 /* Trouble Shooting */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===When can I use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just contact us and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry (you might need to close and reopen your browser). If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I cannot find the installation package for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. == Trouble Shooting == ===I’ve forgot the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that tells me a Python settings file (sitecustomize.py) will be changed. Unfortunately, on my installation there is no such file in that location; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map it to the database columns. If, after thinking it over again, you still want to map literals, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. 2c8b23d04293260315a92f401c192a0e86974a39 295 294 2011-08-15T12:22:22Z JoergHoletschek 3 /* In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that tells me a Python settings file (sitecustomize.py) will be changed. Unfortunately, on my installation the wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===When can I use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just contact us and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry (you might need to close and reopen your browser). If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I cannot find the installation package for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. == Trouble Shooting == ===I’ve forgot the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map it to the database columns. If, after thinking it over again, you still want to map literals, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. 21093b8d445a1d5b84caa3f32a207ec69ce5a179 296 295 2011-08-15T12:23:43Z JoergHoletschek 3 /* In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===When can I use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just contact us and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry (you might need to close and reopen your browser). If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I cannot find the installation package for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. == Trouble Shooting == ===I’ve forgot the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map it to the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. 982f5c4fc64981dfeeb8cf0a5ce107744357f0ab 0 29 268 204 2011-08-11T12:34:04Z JoergHoletschek 3 /* Initial Testing of the Data Source */ wikitext text/x-wiki This tutorial will explain how to set up a mapping for the ABCD2.06 data schema. If you intend to map another schema (e.g. one of the ABCD extension ABCD-EFG, ABCD-DNA or HISPID), you still should take the time to read it. Large portions of the process will be identical. == Access to Biological Collection Data Schema (ABCD) == ABCD in its current version 2.06 is the data schema typically used in conjunction with BioCASe. It is a highly complex XML data schema with about 1,000 elements that is able to store almost every piece of information that can be found in a natural history specimen collection or an observation database. The complexity is due to the fact that it can be used for a wide range of collections/databases – for living and preserved specimens, observations and culture collections, for zoological, botanical, bacterial and viral collections, marine or terrestrial, for herbaria, botanic and zoological gardens. For each of these special types, ABCD features special sections where information specific for this type can be stored. Other sections of ABCD will be shared by all these types, for example for gathering site, identifier or metadata. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a list of commonly used ABCD elements on the Wiki page [[CommonABCD2Concepts]]. == Creating an ABCD Mapping for a Data Source == If you created a data source from a template (e.g. ''abcdmetadata''), a schema mapping for ABCD will be already existent. In this case, you can directly jump to the next section. If you created an empty data source, the section <small>Schemas</small> in the data source configuration overview will be empty: [[File:abcdSchemaMappings.png]] In order to create a new schema mapping, select the desired schema from the list and click <small>Create</small>. For ABCD, choose <small>ABCD_2.06.xml</small> (be careful not to hit one of the ABCD extensions, they look quite similar). After clicking <small>Create</small>, you will be directed to the mapping editor, which will allow you to edit the mappings. == The Mapping Editor == The mapping editor allows you to add, remove and edit ABCD mappings. On top you’ll find some summary information, on the bottom you’ll see the structure of the ABCD mapping: [[File:abcdMappingEditorEmpty.png]] First thing we will do is to set the root table, which is the table that holds the records you want to publish (specimens or observations). That table must hold one record per object to be published, identified by a unique identifier (called ''UnitID''). This identifier doesn't need to be the primary key necessarily, as long as it is unique (if that sentence confuses you, ignore it and just keep in mind that the UnitID must be unique.) Use the drop down box labelled <small>Root table Alias</small> to select the root table of your data model. If your metadata table holds just one record (because all occurrences published share the same metadata) and if it is not linked to the root table by a foreign key, you can specify this table in the drop-down box <small>Static table Alias</small>. The Provider Software will use a natural join to connect the metadata record stored in this table to all occurrences published. If your metadata table is referenced by a foreign key in the root able, you must not declare the metadata table as a static table! So either join the metadata table using a foreign key OR declare it as a static table. When you’ve set the root table (and static table, if you use that), press <small>Save</small>. As with the DB structure setup, this will write your changes to the configuration files, closing the tab without saving will discard your changes! Also, in case you messed something up, you can use the <small>Revert</small> button to restore the state when you’ve saved for the last time. In the mapping editor, the link <small>Overview</small> at the top of the page will take you back to the datasource configuration overview. After creating a mapping, it will appear in the section <small>Schemas</small>. The entry will show the number of mapped elements (0 for now) and the schema namespace. Pressing the trash can symbol will move it to the trash can, which means that your web service does not support this schema anymore; if you did this by mistake, you can restore it by pressing the <small>Restore</small> button. <small>Purge</small> will delete the schema mapping permanently: [[File:abcdSchemaMappings2.png]] You can map several schemas for a given datasource. For example, in addition to the current ABCD2 standard, you could add the deprecated version 1.2 (even though I have no idea why you should do that) or DarwinCore. Just select the appropriate schema in the drop-down list and click <small>Create</small>. Since the namespace of a schema will be used in requests, you cannot create two mappings for the same namespace. Trying this will get you an error message. == Adding Mandatory ABCD Elements == Clicking on a schema in the datasource overview page will take you again to the mapping editor. In the lower half of the page, you will see the basic structure of an ABCD document: [[File:abcdMappingEditorEmptyBottom.png]] <strong>Technical/Content Contact:</strong> These trees hold information on who should be contacted for technical issues or questions concerning the published database, each with name, address, email address and telephone number. <strong>Metadata:</strong> In this sub tree you’ll find all metadata fields, for example title and description of the dataset, ownership information, date last modified, IPR statements etc. <strong>Units:</strong> This part stores the "real" data you want to publish, that means all specimen or observation data. The three elements already shown in the tree are identifiers that make up the so called "Triple ID", which uniquely identifies records within primary biodiversity networks such as BioCASe and GBIF: * ''UnitID'', which is a unique identifier for each record to be published (you can use the catalogue number, for example, or an existing barcode); * ''SourceInstitutionID'', an identifier for the institution publishing the dataset (for example BGBM); * ''SourceID'', an identifier for the dataset (e.g. “Herbarium Berolinense’’). The elements already shown in the mapping editor are printed in red because they are mandatory, which means they are required for an ABCD document. Without them being mapped (to database fields that are non-empty), the Provider Software will not be able to construct valid ABCD documents and will not publish any records. So the first elements that should be mapped are the mandatory fields. To add a mapping, click the <strong>+</strong> symbol next to the field, which will open the mapping editor dialog, where you can select the database table/column you want to map the ABCD element to: [[File:abcdMappingEditorDialog.png]] In this case, the element ''/DataSets/DataSet/Metadata/RevisionData/DateModified'' is mapped to the column ''source_update'' of the ''metadata'' table, which has the data type ''date''. (Remember that table/column retrieval is not supported for all DBMS, so if you’re unlucky, you might need to type in table/column name manually.) <small>Save</small> will accept your settings and close the dialog, <small>Cancel</small> will discard any changes. Instead of mapping an element to a database field, you can map it to a literal. For example, you could enter the technical and content contact information directly as literals in the mapping editor dialog (assuming they’re the same for all records). However, we strongly discourage you from doing this. For one thing, this is just bad style, since it will scatter the information published over different places: Some will be retrieved from the database, some (the literals) will be loaded from the configuration files. Changing the metadata of your service later would then require access to the database and access to the BioCASe configuration – that means the configuration tool password. Be nice to the person who will do these changes and put everything neatly in a database table! The second reason is even more important: Using literals for an element will result in this element to become not searchable, which means it cannot be used in a filter of a Search or a Scan request. Even though this behaviour is discoverable with a Capabilities request (by evaluating the ''searchable'' flag), a potential user of your web service might just not think of that, so it’s a good idea to avoid that in first place. Please use literals sparsely! In the mapping editor window, you can select several source columns and several literals, even a combination of both. The Provider Software will simply concatenate the values retrieved from the database and the literals specified (in the order they’re given). But again: Keep in mind that this will make the element not searchable. A better solution is to do any concatenation operations in a database view. Use the mapping editor window to add mappings for all eight mandatory fields: Click on the <strong>+</strong> symbol next to the element, choose table/column name and the correct data type, then press <small>Save</small>. When you’re done, the page should look similar to the following image. Press <small>Save</small> in the main window to write your changes to the configuration files. [[File:abcdMappingEditorMandatoryElements.png]] == Initial Testing of the Data Source == Once you’ve mapped all mandatory elements, you can do the first test of your BioCASe web service. Even though you haven’t mapped any real information yet, it is a good idea to check the service and solve any problems hitherto before continuing with the mapping. Click on <small>Test mapping!</small> to open the Query Form that can be used for this purpose: [[File:abcdQueryFormEmpty.png]] With the Query Form you can send a BioCASe request to your web service; it will wait for the reply of the web service and display the response document. On top of the page you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ''flora'' being the name of the datasource. Make sure this is correct for your installation! Into the text box below you can enter the request XML text. Because it would be cumbersome to enter the whole BioCASe request, you can use the links below the box to load templates. For now, click on <small>ABCD2 Search</small> to fill the box with a BioCASe Search request for documents in the ABCD2 format (the one we’ve just mapped): <syntaxhighlight> <?xml version='1.0' encoding='UTF-8'?> <request xmlns='http://www.biocase.org/schemas/protocol/1.3'> <header><type>search</type></header> <search> <requestFormat>http://www.tdwg.org/schemas/abcd/2.06</requestFormat> <responseFormat start='0' limit='10'>http://www.tdwg.org/schemas/abcd/2.06</responseFormat> <filter> <like path='/DataSets/DataSet/Units/Unit/Identifications/Identification/Result/ TaxonIdentified/ScientificName/FullScientificNameString'>A*</like> </filter> <count>false</count> </search> </request> </syntaxhighlight> As you can see, there is a filter set on the ''ScientificName'' element. Since we haven’t mapped this element yet, this won’t work. So let’s remove the filter (all lines between <filter> and </filter>), leaving the text box like this: [[File:abcdQueryFormFilled.png]] Pressing <small>Submit</small> will send the request to the web service and – hopefully – display the response document with the correct ABCD records. For the mapping we’ve just created, it should look similar to the following document (use the <strong>–</strong> symbol next to the trees TechnicalContact, ContentContact and Metadata to collapse them): [[File:abcdResponseMandatoryElements.png]] If you get an error, please check out the [[Debugging]] tutorial on how to debug a web service. You should continue the mapping process only after your web service returns results for the basal mapping we’ve just created. == Adding additional ABCD Elements == Once you’ve successfully managed to map the mandatory ABCD elements ("successfully" means the web service returns ABCD documents), you can go on with the mapping process. This is usually an iterative process: You add a handful of elements (for example, country, country code, locality text and coordinates for the gathering site), then you use the Query Form to see if the information ends up where it is supposed to do. This is the reason the Query Form opens up in a separate tab (or window, of you configured your web browser so): In one tab, you use the mapping editor to add new elements; after pressing <small>Save</small> you go to the Query Form tab and see what the response documents look like. Redo this until you’ve mapped all information you want to publish. In order to map new ABCD concepts, tick the check box labelled <small>Show all concepts</small> and press <small>Refresh</small>. The mapping editor will then display the whole ABCD tree, which looks admittedly a bit intimidating. In order to find the concept you’re looking for, use the search function of your browser. If you want to map the country name of the gathering site, for example, hit <small>Ctrl + F</small> (for Firefox, Microsoft IE and Opera) and type in the name of the ABCD element, namely ''country''. The browser will jump to the ''Country'' tree of the ''Gathering'' element, where you’ll find the elements for country name (''Name''), ISO country code (''ISO3166Code'') and the country name in the country’s language (''NameDerived''): [[File:abcdMappingEditorSearch.png|500px]] Continue the sequence mapping new elements/testing until you’ve mapped all information you want to publish. You can have a look at the list of recommend and commonly used ABCD elements on the [[CommonABCD2Concepts]] page. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki]. The following page will show you a sample response ABCD document for a web service with 77 concepts mapped. Of course your documents will look different, because you will probably use different ABCD elements, but still it’s good to get an idea. For easier viewing, you can save the contents of the box into a file with the extension .xml and open that in Firefix or MS Internet Explorer (don't use Safari). You will then be able to collapse the different XML sub trees: [[SampleABCDDocument]]. 248d3ff66f90dc9ba247529eb8f5873d2d1746b2 269 268 2011-08-11T12:35:03Z JoergHoletschek 3 /* Initial Testing of the Data Source */ wikitext text/x-wiki This tutorial will explain how to set up a mapping for the ABCD2.06 data schema. If you intend to map another schema (e.g. one of the ABCD extension ABCD-EFG, ABCD-DNA or HISPID), you still should take the time to read it. Large portions of the process will be identical. == Access to Biological Collection Data Schema (ABCD) == ABCD in its current version 2.06 is the data schema typically used in conjunction with BioCASe. It is a highly complex XML data schema with about 1,000 elements that is able to store almost every piece of information that can be found in a natural history specimen collection or an observation database. The complexity is due to the fact that it can be used for a wide range of collections/databases – for living and preserved specimens, observations and culture collections, for zoological, botanical, bacterial and viral collections, marine or terrestrial, for herbaria, botanic and zoological gardens. For each of these special types, ABCD features special sections where information specific for this type can be stored. Other sections of ABCD will be shared by all these types, for example for gathering site, identifier or metadata. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a list of commonly used ABCD elements on the Wiki page [[CommonABCD2Concepts]]. == Creating an ABCD Mapping for a Data Source == If you created a data source from a template (e.g. ''abcdmetadata''), a schema mapping for ABCD will be already existent. In this case, you can directly jump to the next section. If you created an empty data source, the section <small>Schemas</small> in the data source configuration overview will be empty: [[File:abcdSchemaMappings.png]] In order to create a new schema mapping, select the desired schema from the list and click <small>Create</small>. For ABCD, choose <small>ABCD_2.06.xml</small> (be careful not to hit one of the ABCD extensions, they look quite similar). After clicking <small>Create</small>, you will be directed to the mapping editor, which will allow you to edit the mappings. == The Mapping Editor == The mapping editor allows you to add, remove and edit ABCD mappings. On top you’ll find some summary information, on the bottom you’ll see the structure of the ABCD mapping: [[File:abcdMappingEditorEmpty.png]] First thing we will do is to set the root table, which is the table that holds the records you want to publish (specimens or observations). That table must hold one record per object to be published, identified by a unique identifier (called ''UnitID''). This identifier doesn't need to be the primary key necessarily, as long as it is unique (if that sentence confuses you, ignore it and just keep in mind that the UnitID must be unique.) Use the drop down box labelled <small>Root table Alias</small> to select the root table of your data model. If your metadata table holds just one record (because all occurrences published share the same metadata) and if it is not linked to the root table by a foreign key, you can specify this table in the drop-down box <small>Static table Alias</small>. The Provider Software will use a natural join to connect the metadata record stored in this table to all occurrences published. If your metadata table is referenced by a foreign key in the root able, you must not declare the metadata table as a static table! So either join the metadata table using a foreign key OR declare it as a static table. When you’ve set the root table (and static table, if you use that), press <small>Save</small>. As with the DB structure setup, this will write your changes to the configuration files, closing the tab without saving will discard your changes! Also, in case you messed something up, you can use the <small>Revert</small> button to restore the state when you’ve saved for the last time. In the mapping editor, the link <small>Overview</small> at the top of the page will take you back to the datasource configuration overview. After creating a mapping, it will appear in the section <small>Schemas</small>. The entry will show the number of mapped elements (0 for now) and the schema namespace. Pressing the trash can symbol will move it to the trash can, which means that your web service does not support this schema anymore; if you did this by mistake, you can restore it by pressing the <small>Restore</small> button. <small>Purge</small> will delete the schema mapping permanently: [[File:abcdSchemaMappings2.png]] You can map several schemas for a given datasource. For example, in addition to the current ABCD2 standard, you could add the deprecated version 1.2 (even though I have no idea why you should do that) or DarwinCore. Just select the appropriate schema in the drop-down list and click <small>Create</small>. Since the namespace of a schema will be used in requests, you cannot create two mappings for the same namespace. Trying this will get you an error message. == Adding Mandatory ABCD Elements == Clicking on a schema in the datasource overview page will take you again to the mapping editor. In the lower half of the page, you will see the basic structure of an ABCD document: [[File:abcdMappingEditorEmptyBottom.png]] <strong>Technical/Content Contact:</strong> These trees hold information on who should be contacted for technical issues or questions concerning the published database, each with name, address, email address and telephone number. <strong>Metadata:</strong> In this sub tree you’ll find all metadata fields, for example title and description of the dataset, ownership information, date last modified, IPR statements etc. <strong>Units:</strong> This part stores the "real" data you want to publish, that means all specimen or observation data. The three elements already shown in the tree are identifiers that make up the so called "Triple ID", which uniquely identifies records within primary biodiversity networks such as BioCASe and GBIF: * ''UnitID'', which is a unique identifier for each record to be published (you can use the catalogue number, for example, or an existing barcode); * ''SourceInstitutionID'', an identifier for the institution publishing the dataset (for example BGBM); * ''SourceID'', an identifier for the dataset (e.g. “Herbarium Berolinense’’). The elements already shown in the mapping editor are printed in red because they are mandatory, which means they are required for an ABCD document. Without them being mapped (to database fields that are non-empty), the Provider Software will not be able to construct valid ABCD documents and will not publish any records. So the first elements that should be mapped are the mandatory fields. To add a mapping, click the <strong>+</strong> symbol next to the field, which will open the mapping editor dialog, where you can select the database table/column you want to map the ABCD element to: [[File:abcdMappingEditorDialog.png]] In this case, the element ''/DataSets/DataSet/Metadata/RevisionData/DateModified'' is mapped to the column ''source_update'' of the ''metadata'' table, which has the data type ''date''. (Remember that table/column retrieval is not supported for all DBMS, so if you’re unlucky, you might need to type in table/column name manually.) <small>Save</small> will accept your settings and close the dialog, <small>Cancel</small> will discard any changes. Instead of mapping an element to a database field, you can map it to a literal. For example, you could enter the technical and content contact information directly as literals in the mapping editor dialog (assuming they’re the same for all records). However, we strongly discourage you from doing this. For one thing, this is just bad style, since it will scatter the information published over different places: Some will be retrieved from the database, some (the literals) will be loaded from the configuration files. Changing the metadata of your service later would then require access to the database and access to the BioCASe configuration – that means the configuration tool password. Be nice to the person who will do these changes and put everything neatly in a database table! The second reason is even more important: Using literals for an element will result in this element to become not searchable, which means it cannot be used in a filter of a Search or a Scan request. Even though this behaviour is discoverable with a Capabilities request (by evaluating the ''searchable'' flag), a potential user of your web service might just not think of that, so it’s a good idea to avoid that in first place. Please use literals sparsely! In the mapping editor window, you can select several source columns and several literals, even a combination of both. The Provider Software will simply concatenate the values retrieved from the database and the literals specified (in the order they’re given). But again: Keep in mind that this will make the element not searchable. A better solution is to do any concatenation operations in a database view. Use the mapping editor window to add mappings for all eight mandatory fields: Click on the <strong>+</strong> symbol next to the element, choose table/column name and the correct data type, then press <small>Save</small>. When you’re done, the page should look similar to the following image. Press <small>Save</small> in the main window to write your changes to the configuration files. [[File:abcdMappingEditorMandatoryElements.png]] == Initial Testing of the Data Source == Once you’ve mapped all mandatory elements, you can do the first test of your BioCASe web service. Even though you haven’t mapped any real information yet, it is a good idea to check the service and solve any problems hitherto before continuing with the mapping. Click on <small>Test mapping!</small> to open the Query Form that can be used for this purpose: [[File:abcdQueryFormEmpty.png]] With the Query Form you can send a BioCASe request to your web service; it will wait for the reply of the web service and display the response document. On top of the page you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ''flora'' being the name of the datasource. Make sure this is correct for your installation! Into the text box below you can enter the request XML text. Because it would be cumbersome to enter the whole BioCASe request, you can use the links below the box to load templates. For now, click on <small>ABCD2 Search</small> to fill the box with a BioCASe Search request for documents in the ABCD2 format (the one we’ve just mapped): <syntaxhighlight> <?xml version='1.0' encoding='UTF-8'?> <request xmlns='http://www.biocase.org/schemas/protocol/1.3'> <header><type>search</type></header> <search> <requestFormat>http://www.tdwg.org/schemas/abcd/2.06</requestFormat> <responseFormat start='0' limit='10'>http://www.tdwg.org/schemas/abcd/2.06</responseFormat> <filter> <like path='/DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString'>A*</like> </filter> <count>false</count> </search> </request> </syntaxhighlight> As you can see, there is a filter set on the ''ScientificName'' element. Since we haven’t mapped this element yet, this won’t work. So let’s remove the filter (all lines between <filter> and </filter>), leaving the text box like this: [[File:abcdQueryFormFilled.png]] Pressing <small>Submit</small> will send the request to the web service and – hopefully – display the response document with the correct ABCD records. For the mapping we’ve just created, it should look similar to the following document (use the <strong>–</strong> symbol next to the trees TechnicalContact, ContentContact and Metadata to collapse them): [[File:abcdResponseMandatoryElements.png]] If you get an error, please check out the [[Debugging]] tutorial on how to debug a web service. You should continue the mapping process only after your web service returns results for the basal mapping we’ve just created. == Adding additional ABCD Elements == Once you’ve successfully managed to map the mandatory ABCD elements ("successfully" means the web service returns ABCD documents), you can go on with the mapping process. This is usually an iterative process: You add a handful of elements (for example, country, country code, locality text and coordinates for the gathering site), then you use the Query Form to see if the information ends up where it is supposed to do. This is the reason the Query Form opens up in a separate tab (or window, of you configured your web browser so): In one tab, you use the mapping editor to add new elements; after pressing <small>Save</small> you go to the Query Form tab and see what the response documents look like. Redo this until you’ve mapped all information you want to publish. In order to map new ABCD concepts, tick the check box labelled <small>Show all concepts</small> and press <small>Refresh</small>. The mapping editor will then display the whole ABCD tree, which looks admittedly a bit intimidating. In order to find the concept you’re looking for, use the search function of your browser. If you want to map the country name of the gathering site, for example, hit <small>Ctrl + F</small> (for Firefox, Microsoft IE and Opera) and type in the name of the ABCD element, namely ''country''. The browser will jump to the ''Country'' tree of the ''Gathering'' element, where you’ll find the elements for country name (''Name''), ISO country code (''ISO3166Code'') and the country name in the country’s language (''NameDerived''): [[File:abcdMappingEditorSearch.png|500px]] Continue the sequence mapping new elements/testing until you’ve mapped all information you want to publish. You can have a look at the list of recommend and commonly used ABCD elements on the [[CommonABCD2Concepts]] page. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki]. The following page will show you a sample response ABCD document for a web service with 77 concepts mapped. Of course your documents will look different, because you will probably use different ABCD elements, but still it’s good to get an idea. For easier viewing, you can save the contents of the box into a file with the extension .xml and open that in Firefix or MS Internet Explorer (don't use Safari). You will then be able to collapse the different XML sub trees: [[SampleABCDDocument]]. fc65d4c4d1765241e1bac0a68856c860b5208fc1 0 41 276 252 2011-08-11T12:44:09Z JoergHoletschek 3 /* Header */ wikitext text/x-wiki This tutorial explains how to debug a BioCASe web service and a BioCASe installation in general. If you follow the instructions and don’t manage to get your service running, please [[Special:Contact|contact the BioCASe team]]. == Debugging a BioCASe Web Service == Debugging a BioCASe web service can be done with the manual Query Form. It allows you to load a BioCASe request (Search/Scan) from a template, edit the parameters of the request (e.g. the filter) and will show you the response document, which holds the requested ABCD records and useful debug information. === Opening the Query Form === There are several ways to open the Query Form. On the entry page of a datasource, <small>TEST and DEBUG</small> will open the Query Form: [[File:debugDatasourceEntryPage.png]] In the data source configuration tool, the link to the according query form can be always found in the link list at the top of the page. Moreover, in the mapping editor, the button <small>Test mapping!</small> will open the Query Form in a separate tab: [[File:abcdMappingEditorEmptyTop.png]] === Basic Usage === On top of the Query Form you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ''flora'' being the name of the datasource. Make sure this is the correct access point of the web service you want to debug! [[File:debugQueryformFull.png]] The usage of the Query Form is very straightforward: * Click on one of the links below the text box to load a template, for example for an ABCD2 Scan or ABCD2 Search; * Edit the request parameters; * Click on <small>Submit</small> to get the response document. The section <small>BioCASe protocol filter operators</small> lists the operators that can be used for creating complex filters that combine several filter criteria. More about creating filters can be found in the BioCASe Protocol documentation at http://www.biocase.org/products/protocols/index.shtml. Clicking on the <small>Show</small> link in the <small>Capabilities</small> section will show the Capabilities response in a frame of the Query Form. This is useful, since it shows the namespaces for the schemas supported by the web service (which is required in the request text) and XPaths of mapped ABCD elements (required for creating filters on these elements). By using Copy & Paste you can spare the trouble of entering these monstrous strings of characters. === Changing the Debug Level for a Data Source === Debug info can be set to one of four levels for a BioCASe data source: * <strong>Error</strong> is the highest level will only output errors that prevent the web server from answering a request. * <strong>Warning</strong> will output errors that keep BioCASe from answering a request and warnings that don’t prevent the fulfilling of the request, but should be analysed carefully. For example, they might tell you that records have been dropped because mandatory ABCD elements have not been mapped or were empty for certain records. * <strong>Info</strong> is the default and will add some more information about how the Provider Software processes the request. * <strong>Debug</strong> is the most detailed level and will produce very verbose debug output. It adds very comprehensive information about how the ABCD documents are constructed and is suitable only for experienced users. For most cases, keeping the default ''Info'' level should be sufficient for testing and debugging a web service. For production use, you should set it to ''Error'' later. You can do this in the configuration of a datasource under Settings: [[File:debugDatasourceSettings.png]] == Analyzing the Response Document == Upon clicking <small>Submit</small>, the Query Form will send the request to the web service and wait for the response. It might take some time – up to minutes – until the response document shows up. Amongst others, this depends on * the complexity of the query (especially the filters), * the size of the dataset returned, * the debug level set (“debug” results in very exhaustive debug output), * the performance of the servers running BioCASe and the database, * the network bandwidth. The response document consists of three sections (a full response document can be found [[CommonABCD2Concepts|here]]): [[File:debugResponseDocumentOverview.png]] === Header === This section contains information about the BioCASe installation – operating system, DBMS used, Python/BioCASe version etc. It is intended for a consumer application and won’t be very helpful for debugging: <syntaxhighlight> <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> </syntaxhighlight> === Content === This section makes up the payload of the response document, depending on the type of request: <strong>Capabilities:</strong> Lists all schemas supported by the web service. For each schema, all mapped concepts (identified by their XPaths) are listed with data type and searchable flag. <strong>Scan:</strong> The document will simply list all distinct values for the concept specified in the request. When done for the scientific name, it would look similar to this: <pre> <biocase:content recordCount="1664" recordDropped="0" recordStart="0" totalSearchHits="1664"> <biocase:scan> <biocase:value>Achillea atrata L.</biocase:value> <biocase:value>Achillea millefolium L.</biocase:value> <biocase:value>Achillea nobilis L. ssp. nobilis</biocase:value> <biocase:value>Achillea ptarmica L.</biocase:value> <biocase:value>Acinos alpinus (L.) Moench</biocase:value> <biocase:value>Acinos arvensis (Lam.) Dandy</biocase:value> <biocase:value>Aconitum degenii ssp. paniculatum (Arcang.) Mucher</biocase:value> <biocase:value>Aconitum lycoctonum L. ssp. lycoctonum</biocase:value> <biocase:value>Aconitum napellus L. s.l.</biocase:value> ... </biocase:scan> </biocase:content> </pre> <strong>Search:</strong> For this type of request, the content consists of the ABCD dataset(s). See the the [[CommonABCD2Concepts| ABCD2 example]]. === Diagnostics (Debug Output) === This section is the most interesting part for debugging. If debug level is set to ''Info'' (which is recommended for debugging), it will document the processing of the request by the web service. The output is easy to understand, for example it will contain the SQL statements sent to the database. For a successful search request it could look similar to this: <pre> <biocase:diagnostics> <biocase:diagnostic severity="INFO"> Datasource wrapper FloraExsiccataBavarica requested </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Reading PSF from C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica \provider_setup_file.xml </biocase:diagnostic> <biocase:diagnostic severity="DEBUG"> PSF: PSF=C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\provider_setup_file.xml, recLimit=100, loglevel=20, user=webuser, database=rbg, dbIP=192.168.2.10, dbms=mysql, encoding=latin_1, schemas={u'http://www.tdwg.org/schemas/abcd/2.06': <biocase.wrapper.psf_handler.SupportedSchema instance at 0x011F5B98>}, tablegraph=GRAPH: graph: unit-metadata, +++ ALIAS2TABLE: {u'unit': u'feb3', u'metadata': u'metadata'} </biocase:diagnostic> <biocase:diagnostic severity="INFO"> BioCASe protocol used. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> HTTP parameter 'query' used for building the request. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT DISTINCT unit.id FROM feb3 AS unit WHERE (unit.nameautor LIKE 'Allium %' ) LIMIT 11' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT unit.id, metadata.supplier_organisation, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (40, 41, 42, 43, 44, 45, 46, 47)) ORDER BY unit.id' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> time to execute request is 0.46599984169 </biocase:diagnostic> </biocase:diagnostics> </pre> == Sample Errors and Warnings in the Debug Output == <pre> <biocase:diagnostic severity="ERROR"> No DB connection could be established. Please verify DB state and connection parameters in your provider_setup_file.xml. </biocase:diagnostic> </pre> Well... just what it says. The Provider Software could not connect to the database to be published. Check the database connection settings (server IP, database name, user, and password) and make sure you’ve installed the Python package for the DBMS you’re using (see the <small>Libs test</small> page in <small>Utilities</small>). <pre> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\flora\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="ERROR"> The CMF DB mapping file http://www.tdwg.org/schemas/abcd/2.06 was corrupt or could not be interpreted. </biocase:diagnostic> </pre> The requested schema (identified by the namespace http://www.tdwg.org/schemas/abcd/2.06) is not supported by that service. Create a mapping and map elements. <pre> <biocase:diagnostic severity="ERROR"> The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result /TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts </biocase:diagnostic> </pre> Obvious - happens if you try to use a concept that is not mapped in a scan request or a filter. <pre> <biocase:diagnostic severity="ERROR"> A SQL statement produced an error: SELECT unit.id, metadata.supplier_organisation, metadata.supplier_email, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) ORDER BY unit.id </pre> That could have a number of reasons: One of the table/column names that were set up in the configuration does not exist (because it was renamed or removed) or the credentials used by the BPS do not have sufficient privileges. Simply copy the SQL statement and execute it manually on the database with a regular database client that will show you the detailed error message returned by the DBMS. <pre> <biocase:diagnostic severity="WARNING"> The mandatory child element SourceInstitutionID could not be created [0 existing, but 1 required]. The element /DataSets/DataSet/Units/Unit was also dropped. </biocase:diagnostic> <biocase:diagnostic severity="WARNING"> The mandatory child element Units could not be created [0 existing, but 1 required]. The element /DataSets/DataSet was also dropped. </biocase:diagnostic> </pre> One problem caused the next one in this sequence of warnings: The mandatory ABCD element ''SourceInstitutionID '' was either not mapped or empty; therefore the respective record (''unit'') was dropped. Because this happened for all units in the dataset, all units were dropped, resulting in an empty ''Units'' sub tree. The second warning tells you what happened then: Because the ''Units'' element is mandatory for the ''Dataset'' element, the whole dataset was dropped. As a result, the response ABCD document was empty. == Debugging a BioCASe Installation == In case the debug output of the web service doesn’t help you to find the problem, or if the BPS behaves not as intended without displaying any error messages, you should have a look at the debug logs written by the BPS. To turn on the Debug logs, go to the System Administration page and set <small>Debug logs</small> to <small>True</small>. [[File:instConfigTool.png|500px]] Redo the step that didn’t work and open the ''log'' folder of your BioCASe installation. There are several global log files (pywrapper_request.log, pywrapper_error.log, webapp_debug.log, webapp_error.log) and a debug files per data source (e.g. debug_flora.log). Open these files and see if they can give you a clue to the problem. If you still can’t get it running, [[Special:Contact|contact the BioCASe team]]. 70094ca0116cfe4535319f0a7ff52975772c8343 277 276 2011-08-11T12:44:30Z JoergHoletschek 3 /* Content */ wikitext text/x-wiki This tutorial explains how to debug a BioCASe web service and a BioCASe installation in general. If you follow the instructions and don’t manage to get your service running, please [[Special:Contact|contact the BioCASe team]]. == Debugging a BioCASe Web Service == Debugging a BioCASe web service can be done with the manual Query Form. It allows you to load a BioCASe request (Search/Scan) from a template, edit the parameters of the request (e.g. the filter) and will show you the response document, which holds the requested ABCD records and useful debug information. === Opening the Query Form === There are several ways to open the Query Form. On the entry page of a datasource, <small>TEST and DEBUG</small> will open the Query Form: [[File:debugDatasourceEntryPage.png]] In the data source configuration tool, the link to the according query form can be always found in the link list at the top of the page. Moreover, in the mapping editor, the button <small>Test mapping!</small> will open the Query Form in a separate tab: [[File:abcdMappingEditorEmptyTop.png]] === Basic Usage === On top of the Query Form you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ''flora'' being the name of the datasource. Make sure this is the correct access point of the web service you want to debug! [[File:debugQueryformFull.png]] The usage of the Query Form is very straightforward: * Click on one of the links below the text box to load a template, for example for an ABCD2 Scan or ABCD2 Search; * Edit the request parameters; * Click on <small>Submit</small> to get the response document. The section <small>BioCASe protocol filter operators</small> lists the operators that can be used for creating complex filters that combine several filter criteria. More about creating filters can be found in the BioCASe Protocol documentation at http://www.biocase.org/products/protocols/index.shtml. Clicking on the <small>Show</small> link in the <small>Capabilities</small> section will show the Capabilities response in a frame of the Query Form. This is useful, since it shows the namespaces for the schemas supported by the web service (which is required in the request text) and XPaths of mapped ABCD elements (required for creating filters on these elements). By using Copy & Paste you can spare the trouble of entering these monstrous strings of characters. === Changing the Debug Level for a Data Source === Debug info can be set to one of four levels for a BioCASe data source: * <strong>Error</strong> is the highest level will only output errors that prevent the web server from answering a request. * <strong>Warning</strong> will output errors that keep BioCASe from answering a request and warnings that don’t prevent the fulfilling of the request, but should be analysed carefully. For example, they might tell you that records have been dropped because mandatory ABCD elements have not been mapped or were empty for certain records. * <strong>Info</strong> is the default and will add some more information about how the Provider Software processes the request. * <strong>Debug</strong> is the most detailed level and will produce very verbose debug output. It adds very comprehensive information about how the ABCD documents are constructed and is suitable only for experienced users. For most cases, keeping the default ''Info'' level should be sufficient for testing and debugging a web service. For production use, you should set it to ''Error'' later. You can do this in the configuration of a datasource under Settings: [[File:debugDatasourceSettings.png]] == Analyzing the Response Document == Upon clicking <small>Submit</small>, the Query Form will send the request to the web service and wait for the response. It might take some time – up to minutes – until the response document shows up. Amongst others, this depends on * the complexity of the query (especially the filters), * the size of the dataset returned, * the debug level set (“debug” results in very exhaustive debug output), * the performance of the servers running BioCASe and the database, * the network bandwidth. The response document consists of three sections (a full response document can be found [[CommonABCD2Concepts|here]]): [[File:debugResponseDocumentOverview.png]] === Header === This section contains information about the BioCASe installation – operating system, DBMS used, Python/BioCASe version etc. It is intended for a consumer application and won’t be very helpful for debugging: <syntaxhighlight> <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> </syntaxhighlight> === Content === This section makes up the payload of the response document, depending on the type of request: <strong>Capabilities:</strong> Lists all schemas supported by the web service. For each schema, all mapped concepts (identified by their XPaths) are listed with data type and searchable flag. <strong>Scan:</strong> The document will simply list all distinct values for the concept specified in the request. When done for the scientific name, it would look similar to this: <syntaxhighlight> <biocase:content recordCount="1664" recordDropped="0" recordStart="0" totalSearchHits="1664"> <biocase:scan> <biocase:value>Achillea atrata L.</biocase:value> <biocase:value>Achillea millefolium L.</biocase:value> <biocase:value>Achillea nobilis L. ssp. nobilis</biocase:value> <biocase:value>Achillea ptarmica L.</biocase:value> <biocase:value>Acinos alpinus (L.) Moench</biocase:value> <biocase:value>Acinos arvensis (Lam.) Dandy</biocase:value> <biocase:value>Aconitum degenii ssp. paniculatum (Arcang.) Mucher</biocase:value> <biocase:value>Aconitum lycoctonum L. ssp. lycoctonum</biocase:value> <biocase:value>Aconitum napellus L. s.l.</biocase:value> ... </biocase:scan> </biocase:content> </syntaxhighlight> <strong>Search:</strong> For this type of request, the content consists of the ABCD dataset(s). See the the [[CommonABCD2Concepts| ABCD2 example]]. === Diagnostics (Debug Output) === This section is the most interesting part for debugging. If debug level is set to ''Info'' (which is recommended for debugging), it will document the processing of the request by the web service. The output is easy to understand, for example it will contain the SQL statements sent to the database. For a successful search request it could look similar to this: <pre> <biocase:diagnostics> <biocase:diagnostic severity="INFO"> Datasource wrapper FloraExsiccataBavarica requested </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Reading PSF from C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica \provider_setup_file.xml </biocase:diagnostic> <biocase:diagnostic severity="DEBUG"> PSF: PSF=C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\provider_setup_file.xml, recLimit=100, loglevel=20, user=webuser, database=rbg, dbIP=192.168.2.10, dbms=mysql, encoding=latin_1, schemas={u'http://www.tdwg.org/schemas/abcd/2.06': <biocase.wrapper.psf_handler.SupportedSchema instance at 0x011F5B98>}, tablegraph=GRAPH: graph: unit-metadata, +++ ALIAS2TABLE: {u'unit': u'feb3', u'metadata': u'metadata'} </biocase:diagnostic> <biocase:diagnostic severity="INFO"> BioCASe protocol used. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> HTTP parameter 'query' used for building the request. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT DISTINCT unit.id FROM feb3 AS unit WHERE (unit.nameautor LIKE 'Allium %' ) LIMIT 11' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT unit.id, metadata.supplier_organisation, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (40, 41, 42, 43, 44, 45, 46, 47)) ORDER BY unit.id' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> time to execute request is 0.46599984169 </biocase:diagnostic> </biocase:diagnostics> </pre> == Sample Errors and Warnings in the Debug Output == <pre> <biocase:diagnostic severity="ERROR"> No DB connection could be established. Please verify DB state and connection parameters in your provider_setup_file.xml. </biocase:diagnostic> </pre> Well... just what it says. The Provider Software could not connect to the database to be published. Check the database connection settings (server IP, database name, user, and password) and make sure you’ve installed the Python package for the DBMS you’re using (see the <small>Libs test</small> page in <small>Utilities</small>). <pre> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\flora\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="ERROR"> The CMF DB mapping file http://www.tdwg.org/schemas/abcd/2.06 was corrupt or could not be interpreted. </biocase:diagnostic> </pre> The requested schema (identified by the namespace http://www.tdwg.org/schemas/abcd/2.06) is not supported by that service. Create a mapping and map elements. <pre> <biocase:diagnostic severity="ERROR"> The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result /TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts </biocase:diagnostic> </pre> Obvious - happens if you try to use a concept that is not mapped in a scan request or a filter. <pre> <biocase:diagnostic severity="ERROR"> A SQL statement produced an error: SELECT unit.id, metadata.supplier_organisation, metadata.supplier_email, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) ORDER BY unit.id </pre> That could have a number of reasons: One of the table/column names that were set up in the configuration does not exist (because it was renamed or removed) or the credentials used by the BPS do not have sufficient privileges. Simply copy the SQL statement and execute it manually on the database with a regular database client that will show you the detailed error message returned by the DBMS. <pre> <biocase:diagnostic severity="WARNING"> The mandatory child element SourceInstitutionID could not be created [0 existing, but 1 required]. The element /DataSets/DataSet/Units/Unit was also dropped. </biocase:diagnostic> <biocase:diagnostic severity="WARNING"> The mandatory child element Units could not be created [0 existing, but 1 required]. The element /DataSets/DataSet was also dropped. </biocase:diagnostic> </pre> One problem caused the next one in this sequence of warnings: The mandatory ABCD element ''SourceInstitutionID '' was either not mapped or empty; therefore the respective record (''unit'') was dropped. Because this happened for all units in the dataset, all units were dropped, resulting in an empty ''Units'' sub tree. The second warning tells you what happened then: Because the ''Units'' element is mandatory for the ''Dataset'' element, the whole dataset was dropped. As a result, the response ABCD document was empty. == Debugging a BioCASe Installation == In case the debug output of the web service doesn’t help you to find the problem, or if the BPS behaves not as intended without displaying any error messages, you should have a look at the debug logs written by the BPS. To turn on the Debug logs, go to the System Administration page and set <small>Debug logs</small> to <small>True</small>. [[File:instConfigTool.png|500px]] Redo the step that didn’t work and open the ''log'' folder of your BioCASe installation. There are several global log files (pywrapper_request.log, pywrapper_error.log, webapp_debug.log, webapp_error.log) and a debug files per data source (e.g. debug_flora.log). Open these files and see if they can give you a clue to the problem. If you still can’t get it running, [[Special:Contact|contact the BioCASe team]]. 9842a9f4c5cfe0fb56ce25b65c7cc08805a4a1f4 278 277 2011-08-11T12:44:46Z JoergHoletschek 3 /* Diagnostics (Debug Output) */ wikitext text/x-wiki This tutorial explains how to debug a BioCASe web service and a BioCASe installation in general. If you follow the instructions and don’t manage to get your service running, please [[Special:Contact|contact the BioCASe team]]. == Debugging a BioCASe Web Service == Debugging a BioCASe web service can be done with the manual Query Form. It allows you to load a BioCASe request (Search/Scan) from a template, edit the parameters of the request (e.g. the filter) and will show you the response document, which holds the requested ABCD records and useful debug information. === Opening the Query Form === There are several ways to open the Query Form. On the entry page of a datasource, <small>TEST and DEBUG</small> will open the Query Form: [[File:debugDatasourceEntryPage.png]] In the data source configuration tool, the link to the according query form can be always found in the link list at the top of the page. Moreover, in the mapping editor, the button <small>Test mapping!</small> will open the Query Form in a separate tab: [[File:abcdMappingEditorEmptyTop.png]] === Basic Usage === On top of the Query Form you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ''flora'' being the name of the datasource. Make sure this is the correct access point of the web service you want to debug! [[File:debugQueryformFull.png]] The usage of the Query Form is very straightforward: * Click on one of the links below the text box to load a template, for example for an ABCD2 Scan or ABCD2 Search; * Edit the request parameters; * Click on <small>Submit</small> to get the response document. The section <small>BioCASe protocol filter operators</small> lists the operators that can be used for creating complex filters that combine several filter criteria. More about creating filters can be found in the BioCASe Protocol documentation at http://www.biocase.org/products/protocols/index.shtml. Clicking on the <small>Show</small> link in the <small>Capabilities</small> section will show the Capabilities response in a frame of the Query Form. This is useful, since it shows the namespaces for the schemas supported by the web service (which is required in the request text) and XPaths of mapped ABCD elements (required for creating filters on these elements). By using Copy & Paste you can spare the trouble of entering these monstrous strings of characters. === Changing the Debug Level for a Data Source === Debug info can be set to one of four levels for a BioCASe data source: * <strong>Error</strong> is the highest level will only output errors that prevent the web server from answering a request. * <strong>Warning</strong> will output errors that keep BioCASe from answering a request and warnings that don’t prevent the fulfilling of the request, but should be analysed carefully. For example, they might tell you that records have been dropped because mandatory ABCD elements have not been mapped or were empty for certain records. * <strong>Info</strong> is the default and will add some more information about how the Provider Software processes the request. * <strong>Debug</strong> is the most detailed level and will produce very verbose debug output. It adds very comprehensive information about how the ABCD documents are constructed and is suitable only for experienced users. For most cases, keeping the default ''Info'' level should be sufficient for testing and debugging a web service. For production use, you should set it to ''Error'' later. You can do this in the configuration of a datasource under Settings: [[File:debugDatasourceSettings.png]] == Analyzing the Response Document == Upon clicking <small>Submit</small>, the Query Form will send the request to the web service and wait for the response. It might take some time – up to minutes – until the response document shows up. Amongst others, this depends on * the complexity of the query (especially the filters), * the size of the dataset returned, * the debug level set (“debug” results in very exhaustive debug output), * the performance of the servers running BioCASe and the database, * the network bandwidth. The response document consists of three sections (a full response document can be found [[CommonABCD2Concepts|here]]): [[File:debugResponseDocumentOverview.png]] === Header === This section contains information about the BioCASe installation – operating system, DBMS used, Python/BioCASe version etc. It is intended for a consumer application and won’t be very helpful for debugging: <syntaxhighlight> <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> </syntaxhighlight> === Content === This section makes up the payload of the response document, depending on the type of request: <strong>Capabilities:</strong> Lists all schemas supported by the web service. For each schema, all mapped concepts (identified by their XPaths) are listed with data type and searchable flag. <strong>Scan:</strong> The document will simply list all distinct values for the concept specified in the request. When done for the scientific name, it would look similar to this: <syntaxhighlight> <biocase:content recordCount="1664" recordDropped="0" recordStart="0" totalSearchHits="1664"> <biocase:scan> <biocase:value>Achillea atrata L.</biocase:value> <biocase:value>Achillea millefolium L.</biocase:value> <biocase:value>Achillea nobilis L. ssp. nobilis</biocase:value> <biocase:value>Achillea ptarmica L.</biocase:value> <biocase:value>Acinos alpinus (L.) Moench</biocase:value> <biocase:value>Acinos arvensis (Lam.) Dandy</biocase:value> <biocase:value>Aconitum degenii ssp. paniculatum (Arcang.) Mucher</biocase:value> <biocase:value>Aconitum lycoctonum L. ssp. lycoctonum</biocase:value> <biocase:value>Aconitum napellus L. s.l.</biocase:value> ... </biocase:scan> </biocase:content> </syntaxhighlight> <strong>Search:</strong> For this type of request, the content consists of the ABCD dataset(s). See the the [[CommonABCD2Concepts| ABCD2 example]]. === Diagnostics (Debug Output) === This section is the most interesting part for debugging. If debug level is set to ''Info'' (which is recommended for debugging), it will document the processing of the request by the web service. The output is easy to understand, for example it will contain the SQL statements sent to the database. For a successful search request it could look similar to this: <syntaxhighlight> <biocase:diagnostics> <biocase:diagnostic severity="INFO"> Datasource wrapper FloraExsiccataBavarica requested </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Reading PSF from C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica \provider_setup_file.xml </biocase:diagnostic> <biocase:diagnostic severity="DEBUG"> PSF: PSF=C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\provider_setup_file.xml, recLimit=100, loglevel=20, user=webuser, database=rbg, dbIP=192.168.2.10, dbms=mysql, encoding=latin_1, schemas={u'http://www.tdwg.org/schemas/abcd/2.06': <biocase.wrapper.psf_handler.SupportedSchema instance at 0x011F5B98>}, tablegraph=GRAPH: graph: unit-metadata, +++ ALIAS2TABLE: {u'unit': u'feb3', u'metadata': u'metadata'} </biocase:diagnostic> <biocase:diagnostic severity="INFO"> BioCASe protocol used. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> HTTP parameter 'query' used for building the request. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT DISTINCT unit.id FROM feb3 AS unit WHERE (unit.nameautor LIKE 'Allium %' ) LIMIT 11' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT unit.id, metadata.supplier_organisation, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (40, 41, 42, 43, 44, 45, 46, 47)) ORDER BY unit.id' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> time to execute request is 0.46599984169 </biocase:diagnostic> </biocase:diagnostics> </syntaxhighlight> == Sample Errors and Warnings in the Debug Output == <pre> <biocase:diagnostic severity="ERROR"> No DB connection could be established. Please verify DB state and connection parameters in your provider_setup_file.xml. </biocase:diagnostic> </pre> Well... just what it says. The Provider Software could not connect to the database to be published. Check the database connection settings (server IP, database name, user, and password) and make sure you’ve installed the Python package for the DBMS you’re using (see the <small>Libs test</small> page in <small>Utilities</small>). <pre> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\flora\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="ERROR"> The CMF DB mapping file http://www.tdwg.org/schemas/abcd/2.06 was corrupt or could not be interpreted. </biocase:diagnostic> </pre> The requested schema (identified by the namespace http://www.tdwg.org/schemas/abcd/2.06) is not supported by that service. Create a mapping and map elements. <pre> <biocase:diagnostic severity="ERROR"> The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result /TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts </biocase:diagnostic> </pre> Obvious - happens if you try to use a concept that is not mapped in a scan request or a filter. <pre> <biocase:diagnostic severity="ERROR"> A SQL statement produced an error: SELECT unit.id, metadata.supplier_organisation, metadata.supplier_email, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) ORDER BY unit.id </pre> That could have a number of reasons: One of the table/column names that were set up in the configuration does not exist (because it was renamed or removed) or the credentials used by the BPS do not have sufficient privileges. Simply copy the SQL statement and execute it manually on the database with a regular database client that will show you the detailed error message returned by the DBMS. <pre> <biocase:diagnostic severity="WARNING"> The mandatory child element SourceInstitutionID could not be created [0 existing, but 1 required]. The element /DataSets/DataSet/Units/Unit was also dropped. </biocase:diagnostic> <biocase:diagnostic severity="WARNING"> The mandatory child element Units could not be created [0 existing, but 1 required]. The element /DataSets/DataSet was also dropped. </biocase:diagnostic> </pre> One problem caused the next one in this sequence of warnings: The mandatory ABCD element ''SourceInstitutionID '' was either not mapped or empty; therefore the respective record (''unit'') was dropped. Because this happened for all units in the dataset, all units were dropped, resulting in an empty ''Units'' sub tree. The second warning tells you what happened then: Because the ''Units'' element is mandatory for the ''Dataset'' element, the whole dataset was dropped. As a result, the response ABCD document was empty. == Debugging a BioCASe Installation == In case the debug output of the web service doesn’t help you to find the problem, or if the BPS behaves not as intended without displaying any error messages, you should have a look at the debug logs written by the BPS. To turn on the Debug logs, go to the System Administration page and set <small>Debug logs</small> to <small>True</small>. [[File:instConfigTool.png|500px]] Redo the step that didn’t work and open the ''log'' folder of your BioCASe installation. There are several global log files (pywrapper_request.log, pywrapper_error.log, webapp_debug.log, webapp_error.log) and a debug files per data source (e.g. debug_flora.log). Open these files and see if they can give you a clue to the problem. If you still can’t get it running, [[Special:Contact|contact the BioCASe team]]. c0cbe6caaabba3fe669d851b74ea6ab830dc4459 279 278 2011-08-11T12:47:59Z JoergHoletschek 3 /* Diagnostics (Debug Output) */ wikitext text/x-wiki This tutorial explains how to debug a BioCASe web service and a BioCASe installation in general. If you follow the instructions and don’t manage to get your service running, please [[Special:Contact|contact the BioCASe team]]. == Debugging a BioCASe Web Service == Debugging a BioCASe web service can be done with the manual Query Form. It allows you to load a BioCASe request (Search/Scan) from a template, edit the parameters of the request (e.g. the filter) and will show you the response document, which holds the requested ABCD records and useful debug information. === Opening the Query Form === There are several ways to open the Query Form. On the entry page of a datasource, <small>TEST and DEBUG</small> will open the Query Form: [[File:debugDatasourceEntryPage.png]] In the data source configuration tool, the link to the according query form can be always found in the link list at the top of the page. Moreover, in the mapping editor, the button <small>Test mapping!</small> will open the Query Form in a separate tab: [[File:abcdMappingEditorEmptyTop.png]] === Basic Usage === On top of the Query Form you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ''flora'' being the name of the datasource. Make sure this is the correct access point of the web service you want to debug! [[File:debugQueryformFull.png]] The usage of the Query Form is very straightforward: * Click on one of the links below the text box to load a template, for example for an ABCD2 Scan or ABCD2 Search; * Edit the request parameters; * Click on <small>Submit</small> to get the response document. The section <small>BioCASe protocol filter operators</small> lists the operators that can be used for creating complex filters that combine several filter criteria. More about creating filters can be found in the BioCASe Protocol documentation at http://www.biocase.org/products/protocols/index.shtml. Clicking on the <small>Show</small> link in the <small>Capabilities</small> section will show the Capabilities response in a frame of the Query Form. This is useful, since it shows the namespaces for the schemas supported by the web service (which is required in the request text) and XPaths of mapped ABCD elements (required for creating filters on these elements). By using Copy & Paste you can spare the trouble of entering these monstrous strings of characters. === Changing the Debug Level for a Data Source === Debug info can be set to one of four levels for a BioCASe data source: * <strong>Error</strong> is the highest level will only output errors that prevent the web server from answering a request. * <strong>Warning</strong> will output errors that keep BioCASe from answering a request and warnings that don’t prevent the fulfilling of the request, but should be analysed carefully. For example, they might tell you that records have been dropped because mandatory ABCD elements have not been mapped or were empty for certain records. * <strong>Info</strong> is the default and will add some more information about how the Provider Software processes the request. * <strong>Debug</strong> is the most detailed level and will produce very verbose debug output. It adds very comprehensive information about how the ABCD documents are constructed and is suitable only for experienced users. For most cases, keeping the default ''Info'' level should be sufficient for testing and debugging a web service. For production use, you should set it to ''Error'' later. You can do this in the configuration of a datasource under Settings: [[File:debugDatasourceSettings.png]] == Analyzing the Response Document == Upon clicking <small>Submit</small>, the Query Form will send the request to the web service and wait for the response. It might take some time – up to minutes – until the response document shows up. Amongst others, this depends on * the complexity of the query (especially the filters), * the size of the dataset returned, * the debug level set (“debug” results in very exhaustive debug output), * the performance of the servers running BioCASe and the database, * the network bandwidth. The response document consists of three sections (a full response document can be found [[CommonABCD2Concepts|here]]): [[File:debugResponseDocumentOverview.png]] === Header === This section contains information about the BioCASe installation – operating system, DBMS used, Python/BioCASe version etc. It is intended for a consumer application and won’t be very helpful for debugging: <syntaxhighlight> <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> </syntaxhighlight> === Content === This section makes up the payload of the response document, depending on the type of request: <strong>Capabilities:</strong> Lists all schemas supported by the web service. For each schema, all mapped concepts (identified by their XPaths) are listed with data type and searchable flag. <strong>Scan:</strong> The document will simply list all distinct values for the concept specified in the request. When done for the scientific name, it would look similar to this: <syntaxhighlight> <biocase:content recordCount="1664" recordDropped="0" recordStart="0" totalSearchHits="1664"> <biocase:scan> <biocase:value>Achillea atrata L.</biocase:value> <biocase:value>Achillea millefolium L.</biocase:value> <biocase:value>Achillea nobilis L. ssp. nobilis</biocase:value> <biocase:value>Achillea ptarmica L.</biocase:value> <biocase:value>Acinos alpinus (L.) Moench</biocase:value> <biocase:value>Acinos arvensis (Lam.) Dandy</biocase:value> <biocase:value>Aconitum degenii ssp. paniculatum (Arcang.) Mucher</biocase:value> <biocase:value>Aconitum lycoctonum L. ssp. lycoctonum</biocase:value> <biocase:value>Aconitum napellus L. s.l.</biocase:value> ... </biocase:scan> </biocase:content> </syntaxhighlight> <strong>Search:</strong> For this type of request, the content consists of the ABCD dataset(s). See the the [[CommonABCD2Concepts| ABCD2 example]]. === Diagnostics (Debug Output) === This section is the most interesting part for debugging. If debug level is set to ''Info'' (which is recommended for debugging), it will document the processing of the request by the web service. The output is easy to understand, for example it will contain the SQL statements sent to the database. For a successful search request it could look similar to this: <syntaxhighlight> <biocase:diagnostics> <biocase:diagnostic severity="INFO"> Datasource wrapper FloraExsiccataBavarica requested </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Reading PSF from C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\provider_setup_file.xml </biocase:diagnostic> <biocase:diagnostic severity="DEBUG"> PSF: PSF=C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\provider_setup_file.xml, recLimit=100, loglevel=20, user=webuser, database=rbg, dbIP=192.168.2.10, dbms=mysql, encoding=latin_1, schemas={u'http://www.tdwg.org/schemas/abcd/2.06': <biocase.wrapper.psf_handler.SupportedSchema instance at 0x011F5B98>}, tablegraph=GRAPH: graph: unit-metadata, +++ ALIAS2TABLE: {u'unit': u'feb3', u'metadata': u'metadata'} </biocase:diagnostic> <biocase:diagnostic severity="INFO"> BioCASe protocol used. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> HTTP parameter 'query' used for building the request. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT DISTINCT unit.id FROM feb3 AS unit WHERE (unit.nameautor LIKE 'Allium %' ) LIMIT 11' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT unit.id, metadata.supplier_organisation, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (40, 41, 42, 43, 44, 45, 46, 47)) ORDER BY unit.id' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> time to execute request is 0.46599984169 </biocase:diagnostic> </biocase:diagnostics> </syntaxhighlight> == Sample Errors and Warnings in the Debug Output == <pre> <biocase:diagnostic severity="ERROR"> No DB connection could be established. Please verify DB state and connection parameters in your provider_setup_file.xml. </biocase:diagnostic> </pre> Well... just what it says. The Provider Software could not connect to the database to be published. Check the database connection settings (server IP, database name, user, and password) and make sure you’ve installed the Python package for the DBMS you’re using (see the <small>Libs test</small> page in <small>Utilities</small>). <pre> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\flora\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="ERROR"> The CMF DB mapping file http://www.tdwg.org/schemas/abcd/2.06 was corrupt or could not be interpreted. </biocase:diagnostic> </pre> The requested schema (identified by the namespace http://www.tdwg.org/schemas/abcd/2.06) is not supported by that service. Create a mapping and map elements. <pre> <biocase:diagnostic severity="ERROR"> The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result /TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts </biocase:diagnostic> </pre> Obvious - happens if you try to use a concept that is not mapped in a scan request or a filter. <pre> <biocase:diagnostic severity="ERROR"> A SQL statement produced an error: SELECT unit.id, metadata.supplier_organisation, metadata.supplier_email, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) ORDER BY unit.id </pre> That could have a number of reasons: One of the table/column names that were set up in the configuration does not exist (because it was renamed or removed) or the credentials used by the BPS do not have sufficient privileges. Simply copy the SQL statement and execute it manually on the database with a regular database client that will show you the detailed error message returned by the DBMS. <pre> <biocase:diagnostic severity="WARNING"> The mandatory child element SourceInstitutionID could not be created [0 existing, but 1 required]. The element /DataSets/DataSet/Units/Unit was also dropped. </biocase:diagnostic> <biocase:diagnostic severity="WARNING"> The mandatory child element Units could not be created [0 existing, but 1 required]. The element /DataSets/DataSet was also dropped. </biocase:diagnostic> </pre> One problem caused the next one in this sequence of warnings: The mandatory ABCD element ''SourceInstitutionID '' was either not mapped or empty; therefore the respective record (''unit'') was dropped. Because this happened for all units in the dataset, all units were dropped, resulting in an empty ''Units'' sub tree. The second warning tells you what happened then: Because the ''Units'' element is mandatory for the ''Dataset'' element, the whole dataset was dropped. As a result, the response ABCD document was empty. == Debugging a BioCASe Installation == In case the debug output of the web service doesn’t help you to find the problem, or if the BPS behaves not as intended without displaying any error messages, you should have a look at the debug logs written by the BPS. To turn on the Debug logs, go to the System Administration page and set <small>Debug logs</small> to <small>True</small>. [[File:instConfigTool.png|500px]] Redo the step that didn’t work and open the ''log'' folder of your BioCASe installation. There are several global log files (pywrapper_request.log, pywrapper_error.log, webapp_debug.log, webapp_error.log) and a debug files per data source (e.g. debug_flora.log). Open these files and see if they can give you a clue to the problem. If you still can’t get it running, [[Special:Contact|contact the BioCASe team]]. 2e4a2db3fd5ce04a237017c5bdaa18eae04fd171 280 279 2011-08-11T12:48:43Z JoergHoletschek 3 /* Sample Errors and Warnings in the Debug Output */ wikitext text/x-wiki This tutorial explains how to debug a BioCASe web service and a BioCASe installation in general. If you follow the instructions and don’t manage to get your service running, please [[Special:Contact|contact the BioCASe team]]. == Debugging a BioCASe Web Service == Debugging a BioCASe web service can be done with the manual Query Form. It allows you to load a BioCASe request (Search/Scan) from a template, edit the parameters of the request (e.g. the filter) and will show you the response document, which holds the requested ABCD records and useful debug information. === Opening the Query Form === There are several ways to open the Query Form. On the entry page of a datasource, <small>TEST and DEBUG</small> will open the Query Form: [[File:debugDatasourceEntryPage.png]] In the data source configuration tool, the link to the according query form can be always found in the link list at the top of the page. Moreover, in the mapping editor, the button <small>Test mapping!</small> will open the Query Form in a separate tab: [[File:abcdMappingEditorEmptyTop.png]] === Basic Usage === On top of the Query Form you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ''flora'' being the name of the datasource. Make sure this is the correct access point of the web service you want to debug! [[File:debugQueryformFull.png]] The usage of the Query Form is very straightforward: * Click on one of the links below the text box to load a template, for example for an ABCD2 Scan or ABCD2 Search; * Edit the request parameters; * Click on <small>Submit</small> to get the response document. The section <small>BioCASe protocol filter operators</small> lists the operators that can be used for creating complex filters that combine several filter criteria. More about creating filters can be found in the BioCASe Protocol documentation at http://www.biocase.org/products/protocols/index.shtml. Clicking on the <small>Show</small> link in the <small>Capabilities</small> section will show the Capabilities response in a frame of the Query Form. This is useful, since it shows the namespaces for the schemas supported by the web service (which is required in the request text) and XPaths of mapped ABCD elements (required for creating filters on these elements). By using Copy & Paste you can spare the trouble of entering these monstrous strings of characters. === Changing the Debug Level for a Data Source === Debug info can be set to one of four levels for a BioCASe data source: * <strong>Error</strong> is the highest level will only output errors that prevent the web server from answering a request. * <strong>Warning</strong> will output errors that keep BioCASe from answering a request and warnings that don’t prevent the fulfilling of the request, but should be analysed carefully. For example, they might tell you that records have been dropped because mandatory ABCD elements have not been mapped or were empty for certain records. * <strong>Info</strong> is the default and will add some more information about how the Provider Software processes the request. * <strong>Debug</strong> is the most detailed level and will produce very verbose debug output. It adds very comprehensive information about how the ABCD documents are constructed and is suitable only for experienced users. For most cases, keeping the default ''Info'' level should be sufficient for testing and debugging a web service. For production use, you should set it to ''Error'' later. You can do this in the configuration of a datasource under Settings: [[File:debugDatasourceSettings.png]] == Analyzing the Response Document == Upon clicking <small>Submit</small>, the Query Form will send the request to the web service and wait for the response. It might take some time – up to minutes – until the response document shows up. Amongst others, this depends on * the complexity of the query (especially the filters), * the size of the dataset returned, * the debug level set (“debug” results in very exhaustive debug output), * the performance of the servers running BioCASe and the database, * the network bandwidth. The response document consists of three sections (a full response document can be found [[CommonABCD2Concepts|here]]): [[File:debugResponseDocumentOverview.png]] === Header === This section contains information about the BioCASe installation – operating system, DBMS used, Python/BioCASe version etc. It is intended for a consumer application and won’t be very helpful for debugging: <syntaxhighlight> <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> </syntaxhighlight> === Content === This section makes up the payload of the response document, depending on the type of request: <strong>Capabilities:</strong> Lists all schemas supported by the web service. For each schema, all mapped concepts (identified by their XPaths) are listed with data type and searchable flag. <strong>Scan:</strong> The document will simply list all distinct values for the concept specified in the request. When done for the scientific name, it would look similar to this: <syntaxhighlight> <biocase:content recordCount="1664" recordDropped="0" recordStart="0" totalSearchHits="1664"> <biocase:scan> <biocase:value>Achillea atrata L.</biocase:value> <biocase:value>Achillea millefolium L.</biocase:value> <biocase:value>Achillea nobilis L. ssp. nobilis</biocase:value> <biocase:value>Achillea ptarmica L.</biocase:value> <biocase:value>Acinos alpinus (L.) Moench</biocase:value> <biocase:value>Acinos arvensis (Lam.) Dandy</biocase:value> <biocase:value>Aconitum degenii ssp. paniculatum (Arcang.) Mucher</biocase:value> <biocase:value>Aconitum lycoctonum L. ssp. lycoctonum</biocase:value> <biocase:value>Aconitum napellus L. s.l.</biocase:value> ... </biocase:scan> </biocase:content> </syntaxhighlight> <strong>Search:</strong> For this type of request, the content consists of the ABCD dataset(s). See the the [[CommonABCD2Concepts| ABCD2 example]]. === Diagnostics (Debug Output) === This section is the most interesting part for debugging. If debug level is set to ''Info'' (which is recommended for debugging), it will document the processing of the request by the web service. The output is easy to understand, for example it will contain the SQL statements sent to the database. For a successful search request it could look similar to this: <syntaxhighlight> <biocase:diagnostics> <biocase:diagnostic severity="INFO"> Datasource wrapper FloraExsiccataBavarica requested </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Reading PSF from C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\provider_setup_file.xml </biocase:diagnostic> <biocase:diagnostic severity="DEBUG"> PSF: PSF=C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\provider_setup_file.xml, recLimit=100, loglevel=20, user=webuser, database=rbg, dbIP=192.168.2.10, dbms=mysql, encoding=latin_1, schemas={u'http://www.tdwg.org/schemas/abcd/2.06': <biocase.wrapper.psf_handler.SupportedSchema instance at 0x011F5B98>}, tablegraph=GRAPH: graph: unit-metadata, +++ ALIAS2TABLE: {u'unit': u'feb3', u'metadata': u'metadata'} </biocase:diagnostic> <biocase:diagnostic severity="INFO"> BioCASe protocol used. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> HTTP parameter 'query' used for building the request. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT DISTINCT unit.id FROM feb3 AS unit WHERE (unit.nameautor LIKE 'Allium %' ) LIMIT 11' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT unit.id, metadata.supplier_organisation, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (40, 41, 42, 43, 44, 45, 46, 47)) ORDER BY unit.id' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> time to execute request is 0.46599984169 </biocase:diagnostic> </biocase:diagnostics> </syntaxhighlight> == Sample Errors and Warnings in the Debug Output == <syntaxhighlight> <biocase:diagnostic severity="ERROR"> No DB connection could be established. Please verify DB state and connection parameters in your provider_setup_file.xml. </biocase:diagnostic> </syntaxhighlight> Well... just what it says. The Provider Software could not connect to the database to be published. Check the database connection settings (server IP, database name, user, and password) and make sure you’ve installed the Python package for the DBMS you’re using (see the <small>Libs test</small> page in <small>Utilities</small>). <syntaxhighlight> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\flora\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="ERROR"> The CMF DB mapping file http://www.tdwg.org/schemas/abcd/2.06 was corrupt or could not be interpreted. </biocase:diagnostic> </syntaxhighlight> The requested schema (identified by the namespace http://www.tdwg.org/schemas/abcd/2.06) is not supported by that service. Create a mapping and map elements. <syntaxhighlight> <biocase:diagnostic severity="ERROR"> The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result /TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts </biocase:diagnostic> </syntaxhighlight> Obvious - happens if you try to use a concept that is not mapped in a scan request or a filter. <syntaxhighlight> <biocase:diagnostic severity="ERROR"> A SQL statement produced an error: SELECT unit.id, metadata.supplier_organisation, metadata.supplier_email, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) ORDER BY unit.id </syntaxhighlight> That could have a number of reasons: One of the table/column names that were set up in the configuration does not exist (because it was renamed or removed) or the credentials used by the BPS do not have sufficient privileges. Simply copy the SQL statement and execute it manually on the database with a regular database client that will show you the detailed error message returned by the DBMS. <syntaxhighlight> <biocase:diagnostic severity="WARNING"> The mandatory child element SourceInstitutionID could not be created [0 existing, but 1 required]. The element /DataSets/DataSet/Units/Unit was also dropped. </biocase:diagnostic> <biocase:diagnostic severity="WARNING"> The mandatory child element Units could not be created [0 existing, but 1 required]. The element /DataSets/DataSet was also dropped. </biocase:diagnostic> </syntaxhighlight> One problem caused the next one in this sequence of warnings: The mandatory ABCD element ''SourceInstitutionID '' was either not mapped or empty; therefore the respective record (''unit'') was dropped. Because this happened for all units in the dataset, all units were dropped, resulting in an empty ''Units'' sub tree. The second warning tells you what happened then: Because the ''Units'' element is mandatory for the ''Dataset'' element, the whole dataset was dropped. As a result, the response ABCD document was empty. == Debugging a BioCASe Installation == In case the debug output of the web service doesn’t help you to find the problem, or if the BPS behaves not as intended without displaying any error messages, you should have a look at the debug logs written by the BPS. To turn on the Debug logs, go to the System Administration page and set <small>Debug logs</small> to <small>True</small>. [[File:instConfigTool.png|500px]] Redo the step that didn’t work and open the ''log'' folder of your BioCASe installation. There are several global log files (pywrapper_request.log, pywrapper_error.log, webapp_debug.log, webapp_error.log) and a debug files per data source (e.g. debug_flora.log). Open these files and see if they can give you a clue to the problem. If you still can’t get it running, [[Special:Contact|contact the BioCASe team]]. 155468c4a02b049a220568c518899d1381264cb0 281 280 2011-08-11T12:49:24Z JoergHoletschek 3 /* Sample Errors and Warnings in the Debug Output */ wikitext text/x-wiki This tutorial explains how to debug a BioCASe web service and a BioCASe installation in general. If you follow the instructions and don’t manage to get your service running, please [[Special:Contact|contact the BioCASe team]]. == Debugging a BioCASe Web Service == Debugging a BioCASe web service can be done with the manual Query Form. It allows you to load a BioCASe request (Search/Scan) from a template, edit the parameters of the request (e.g. the filter) and will show you the response document, which holds the requested ABCD records and useful debug information. === Opening the Query Form === There are several ways to open the Query Form. On the entry page of a datasource, <small>TEST and DEBUG</small> will open the Query Form: [[File:debugDatasourceEntryPage.png]] In the data source configuration tool, the link to the according query form can be always found in the link list at the top of the page. Moreover, in the mapping editor, the button <small>Test mapping!</small> will open the Query Form in a separate tab: [[File:abcdMappingEditorEmptyTop.png]] === Basic Usage === On top of the Query Form you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ''flora'' being the name of the datasource. Make sure this is the correct access point of the web service you want to debug! [[File:debugQueryformFull.png]] The usage of the Query Form is very straightforward: * Click on one of the links below the text box to load a template, for example for an ABCD2 Scan or ABCD2 Search; * Edit the request parameters; * Click on <small>Submit</small> to get the response document. The section <small>BioCASe protocol filter operators</small> lists the operators that can be used for creating complex filters that combine several filter criteria. More about creating filters can be found in the BioCASe Protocol documentation at http://www.biocase.org/products/protocols/index.shtml. Clicking on the <small>Show</small> link in the <small>Capabilities</small> section will show the Capabilities response in a frame of the Query Form. This is useful, since it shows the namespaces for the schemas supported by the web service (which is required in the request text) and XPaths of mapped ABCD elements (required for creating filters on these elements). By using Copy & Paste you can spare the trouble of entering these monstrous strings of characters. === Changing the Debug Level for a Data Source === Debug info can be set to one of four levels for a BioCASe data source: * <strong>Error</strong> is the highest level will only output errors that prevent the web server from answering a request. * <strong>Warning</strong> will output errors that keep BioCASe from answering a request and warnings that don’t prevent the fulfilling of the request, but should be analysed carefully. For example, they might tell you that records have been dropped because mandatory ABCD elements have not been mapped or were empty for certain records. * <strong>Info</strong> is the default and will add some more information about how the Provider Software processes the request. * <strong>Debug</strong> is the most detailed level and will produce very verbose debug output. It adds very comprehensive information about how the ABCD documents are constructed and is suitable only for experienced users. For most cases, keeping the default ''Info'' level should be sufficient for testing and debugging a web service. For production use, you should set it to ''Error'' later. You can do this in the configuration of a datasource under Settings: [[File:debugDatasourceSettings.png]] == Analyzing the Response Document == Upon clicking <small>Submit</small>, the Query Form will send the request to the web service and wait for the response. It might take some time – up to minutes – until the response document shows up. Amongst others, this depends on * the complexity of the query (especially the filters), * the size of the dataset returned, * the debug level set (“debug” results in very exhaustive debug output), * the performance of the servers running BioCASe and the database, * the network bandwidth. The response document consists of three sections (a full response document can be found [[CommonABCD2Concepts|here]]): [[File:debugResponseDocumentOverview.png]] === Header === This section contains information about the BioCASe installation – operating system, DBMS used, Python/BioCASe version etc. It is intended for a consumer application and won’t be very helpful for debugging: <syntaxhighlight> <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> </syntaxhighlight> === Content === This section makes up the payload of the response document, depending on the type of request: <strong>Capabilities:</strong> Lists all schemas supported by the web service. For each schema, all mapped concepts (identified by their XPaths) are listed with data type and searchable flag. <strong>Scan:</strong> The document will simply list all distinct values for the concept specified in the request. When done for the scientific name, it would look similar to this: <syntaxhighlight> <biocase:content recordCount="1664" recordDropped="0" recordStart="0" totalSearchHits="1664"> <biocase:scan> <biocase:value>Achillea atrata L.</biocase:value> <biocase:value>Achillea millefolium L.</biocase:value> <biocase:value>Achillea nobilis L. ssp. nobilis</biocase:value> <biocase:value>Achillea ptarmica L.</biocase:value> <biocase:value>Acinos alpinus (L.) Moench</biocase:value> <biocase:value>Acinos arvensis (Lam.) Dandy</biocase:value> <biocase:value>Aconitum degenii ssp. paniculatum (Arcang.) Mucher</biocase:value> <biocase:value>Aconitum lycoctonum L. ssp. lycoctonum</biocase:value> <biocase:value>Aconitum napellus L. s.l.</biocase:value> ... </biocase:scan> </biocase:content> </syntaxhighlight> <strong>Search:</strong> For this type of request, the content consists of the ABCD dataset(s). See the the [[CommonABCD2Concepts| ABCD2 example]]. === Diagnostics (Debug Output) === This section is the most interesting part for debugging. If debug level is set to ''Info'' (which is recommended for debugging), it will document the processing of the request by the web service. The output is easy to understand, for example it will contain the SQL statements sent to the database. For a successful search request it could look similar to this: <syntaxhighlight> <biocase:diagnostics> <biocase:diagnostic severity="INFO"> Datasource wrapper FloraExsiccataBavarica requested </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Reading PSF from C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\provider_setup_file.xml </biocase:diagnostic> <biocase:diagnostic severity="DEBUG"> PSF: PSF=C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\provider_setup_file.xml, recLimit=100, loglevel=20, user=webuser, database=rbg, dbIP=192.168.2.10, dbms=mysql, encoding=latin_1, schemas={u'http://www.tdwg.org/schemas/abcd/2.06': <biocase.wrapper.psf_handler.SupportedSchema instance at 0x011F5B98>}, tablegraph=GRAPH: graph: unit-metadata, +++ ALIAS2TABLE: {u'unit': u'feb3', u'metadata': u'metadata'} </biocase:diagnostic> <biocase:diagnostic severity="INFO"> BioCASe protocol used. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> HTTP parameter 'query' used for building the request. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT DISTINCT unit.id FROM feb3 AS unit WHERE (unit.nameautor LIKE 'Allium %' ) LIMIT 11' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT unit.id, metadata.supplier_organisation, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (40, 41, 42, 43, 44, 45, 46, 47)) ORDER BY unit.id' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> time to execute request is 0.46599984169 </biocase:diagnostic> </biocase:diagnostics> </syntaxhighlight> == Sample Errors and Warnings in the Debug Output == <syntaxhighlight> <biocase:diagnostic severity="ERROR"> No DB connection could be established. Please verify DB state and connection parameters in your provider_setup_file.xml. </biocase:diagnostic> </syntaxhighlight> Well... just what it says. The Provider Software could not connect to the database to be published. Check the database connection settings (server IP, database name, user, and password) and make sure you’ve installed the Python package for the DBMS you’re using (see the <small>Libs test</small> page in <small>Utilities</small>). <syntaxhighlight> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\flora\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="ERROR"> The CMF DB mapping file http://www.tdwg.org/schemas/abcd/2.06 was corrupt or could not be interpreted. </biocase:diagnostic> </syntaxhighlight> The requested schema (identified by the namespace http://www.tdwg.org/schemas/abcd/2.06) is not supported by that service. Create a mapping and map elements. <syntaxhighlight> <biocase:diagnostic severity="ERROR"> The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result /TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts </biocase:diagnostic> </syntaxhighlight> Obvious - happens if you try to use a concept that is not mapped in a scan request or a filter. <syntaxhighlight> <biocase:diagnostic severity="ERROR"> A SQL statement produced an error: SELECT unit.id, metadata.supplier_organisation, metadata.supplier_email, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) ORDER BY unit.id </biocase:diagnostic> </syntaxhighlight> That could have a number of reasons: One of the table/column names that were set up in the configuration does not exist (because it was renamed or removed) or the credentials used by the BPS do not have sufficient privileges. Simply copy the SQL statement and execute it manually on the database with a regular database client that will show you the detailed error message returned by the DBMS. <syntaxhighlight> <biocase:diagnostic severity="WARNING"> The mandatory child element SourceInstitutionID could not be created [0 existing, but 1 required]. The element /DataSets/DataSet/Units/Unit was also dropped. </biocase:diagnostic> <biocase:diagnostic severity="WARNING"> The mandatory child element Units could not be created [0 existing, but 1 required]. The element /DataSets/DataSet was also dropped. </biocase:diagnostic> </syntaxhighlight> One problem caused the next one in this sequence of warnings: The mandatory ABCD element ''SourceInstitutionID '' was either not mapped or empty; therefore the respective record (''unit'') was dropped. Because this happened for all units in the dataset, all units were dropped, resulting in an empty ''Units'' sub tree. The second warning tells you what happened then: Because the ''Units'' element is mandatory for the ''Dataset'' element, the whole dataset was dropped. As a result, the response ABCD document was empty. == Debugging a BioCASe Installation == In case the debug output of the web service doesn’t help you to find the problem, or if the BPS behaves not as intended without displaying any error messages, you should have a look at the debug logs written by the BPS. To turn on the Debug logs, go to the System Administration page and set <small>Debug logs</small> to <small>True</small>. [[File:instConfigTool.png|500px]] Redo the step that didn’t work and open the ''log'' folder of your BioCASe installation. There are several global log files (pywrapper_request.log, pywrapper_error.log, webapp_debug.log, webapp_error.log) and a debug files per data source (e.g. debug_flora.log). Open these files and see if they can give you a clue to the problem. If you still can’t get it running, [[Special:Contact|contact the BioCASe team]]. 1c50233629e38c3ae03eeb0befee92dae277ab26 283 281 2011-08-11T12:51:50Z JoergHoletschek 3 /* Diagnostics (Debug Output) */ wikitext text/x-wiki This tutorial explains how to debug a BioCASe web service and a BioCASe installation in general. If you follow the instructions and don’t manage to get your service running, please [[Special:Contact|contact the BioCASe team]]. == Debugging a BioCASe Web Service == Debugging a BioCASe web service can be done with the manual Query Form. It allows you to load a BioCASe request (Search/Scan) from a template, edit the parameters of the request (e.g. the filter) and will show you the response document, which holds the requested ABCD records and useful debug information. === Opening the Query Form === There are several ways to open the Query Form. On the entry page of a datasource, <small>TEST and DEBUG</small> will open the Query Form: [[File:debugDatasourceEntryPage.png]] In the data source configuration tool, the link to the according query form can be always found in the link list at the top of the page. Moreover, in the mapping editor, the button <small>Test mapping!</small> will open the Query Form in a separate tab: [[File:abcdMappingEditorEmptyTop.png]] === Basic Usage === On top of the Query Form you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ''flora'' being the name of the datasource. Make sure this is the correct access point of the web service you want to debug! [[File:debugQueryformFull.png]] The usage of the Query Form is very straightforward: * Click on one of the links below the text box to load a template, for example for an ABCD2 Scan or ABCD2 Search; * Edit the request parameters; * Click on <small>Submit</small> to get the response document. The section <small>BioCASe protocol filter operators</small> lists the operators that can be used for creating complex filters that combine several filter criteria. More about creating filters can be found in the BioCASe Protocol documentation at http://www.biocase.org/products/protocols/index.shtml. Clicking on the <small>Show</small> link in the <small>Capabilities</small> section will show the Capabilities response in a frame of the Query Form. This is useful, since it shows the namespaces for the schemas supported by the web service (which is required in the request text) and XPaths of mapped ABCD elements (required for creating filters on these elements). By using Copy & Paste you can spare the trouble of entering these monstrous strings of characters. === Changing the Debug Level for a Data Source === Debug info can be set to one of four levels for a BioCASe data source: * <strong>Error</strong> is the highest level will only output errors that prevent the web server from answering a request. * <strong>Warning</strong> will output errors that keep BioCASe from answering a request and warnings that don’t prevent the fulfilling of the request, but should be analysed carefully. For example, they might tell you that records have been dropped because mandatory ABCD elements have not been mapped or were empty for certain records. * <strong>Info</strong> is the default and will add some more information about how the Provider Software processes the request. * <strong>Debug</strong> is the most detailed level and will produce very verbose debug output. It adds very comprehensive information about how the ABCD documents are constructed and is suitable only for experienced users. For most cases, keeping the default ''Info'' level should be sufficient for testing and debugging a web service. For production use, you should set it to ''Error'' later. You can do this in the configuration of a datasource under Settings: [[File:debugDatasourceSettings.png]] == Analyzing the Response Document == Upon clicking <small>Submit</small>, the Query Form will send the request to the web service and wait for the response. It might take some time – up to minutes – until the response document shows up. Amongst others, this depends on * the complexity of the query (especially the filters), * the size of the dataset returned, * the debug level set (“debug” results in very exhaustive debug output), * the performance of the servers running BioCASe and the database, * the network bandwidth. The response document consists of three sections (a full response document can be found [[CommonABCD2Concepts|here]]): [[File:debugResponseDocumentOverview.png]] === Header === This section contains information about the BioCASe installation – operating system, DBMS used, Python/BioCASe version etc. It is intended for a consumer application and won’t be very helpful for debugging: <syntaxhighlight> <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> </syntaxhighlight> === Content === This section makes up the payload of the response document, depending on the type of request: <strong>Capabilities:</strong> Lists all schemas supported by the web service. For each schema, all mapped concepts (identified by their XPaths) are listed with data type and searchable flag. <strong>Scan:</strong> The document will simply list all distinct values for the concept specified in the request. When done for the scientific name, it would look similar to this: <syntaxhighlight> <biocase:content recordCount="1664" recordDropped="0" recordStart="0" totalSearchHits="1664"> <biocase:scan> <biocase:value>Achillea atrata L.</biocase:value> <biocase:value>Achillea millefolium L.</biocase:value> <biocase:value>Achillea nobilis L. ssp. nobilis</biocase:value> <biocase:value>Achillea ptarmica L.</biocase:value> <biocase:value>Acinos alpinus (L.) Moench</biocase:value> <biocase:value>Acinos arvensis (Lam.) Dandy</biocase:value> <biocase:value>Aconitum degenii ssp. paniculatum (Arcang.) Mucher</biocase:value> <biocase:value>Aconitum lycoctonum L. ssp. lycoctonum</biocase:value> <biocase:value>Aconitum napellus L. s.l.</biocase:value> ... </biocase:scan> </biocase:content> </syntaxhighlight> <strong>Search:</strong> For this type of request, the content consists of the ABCD dataset(s). See the the [[CommonABCD2Concepts| ABCD2 example]]. === Diagnostics (Debug Output) === This section is the most interesting part for debugging. If debug level is set to ''Info'' (which is recommended for debugging), it will document the processing of the request by the web service. The output is easy to understand, for example it will contain the SQL statements sent to the database. For a successful search request it could look similar to this: <syntaxhighlight> <biocase:diagnostics> <biocase:diagnostic severity="INFO"> Datasource wrapper FloraExsiccataBavarica requested </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Reading PSF from C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica \provider_setup_file.xml </biocase:diagnostic> <biocase:diagnostic severity="DEBUG"> PSF: PSF=C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\provider_setup_file.xml, recLimit=100, loglevel=20, user=webuser, database=rbg, dbIP=192.168.2.10, dbms=mysql, encoding=latin_1, schemas={u'http://www.tdwg.org/schemas/abcd/2.06': <biocase.wrapper.psf_handler.SupportedSchema instance at 0x011F5B98>}, tablegraph=GRAPH: graph: unit-metadata, +++ ALIAS2TABLE: {u'unit': u'feb3', u'metadata': u'metadata'} </biocase:diagnostic> <biocase:diagnostic severity="INFO"> BioCASe protocol used. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> HTTP parameter 'query' used for building the request. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT DISTINCT unit.id FROM feb3 AS unit WHERE (unit.nameautor LIKE 'Allium %' ) LIMIT 11' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT unit.id, metadata.supplier_organisation, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (40, 41, 42, 43, 44, 45, 46, 47)) ORDER BY unit.id' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> time to execute request is 0.46599984169 </biocase:diagnostic> </biocase:diagnostics> </syntaxhighlight> == Sample Errors and Warnings in the Debug Output == <syntaxhighlight> <biocase:diagnostic severity="ERROR"> No DB connection could be established. Please verify DB state and connection parameters in your provider_setup_file.xml. </biocase:diagnostic> </syntaxhighlight> Well... just what it says. The Provider Software could not connect to the database to be published. Check the database connection settings (server IP, database name, user, and password) and make sure you’ve installed the Python package for the DBMS you’re using (see the <small>Libs test</small> page in <small>Utilities</small>). <syntaxhighlight> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\flora\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="ERROR"> The CMF DB mapping file http://www.tdwg.org/schemas/abcd/2.06 was corrupt or could not be interpreted. </biocase:diagnostic> </syntaxhighlight> The requested schema (identified by the namespace http://www.tdwg.org/schemas/abcd/2.06) is not supported by that service. Create a mapping and map elements. <syntaxhighlight> <biocase:diagnostic severity="ERROR"> The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result /TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts </biocase:diagnostic> </syntaxhighlight> Obvious - happens if you try to use a concept that is not mapped in a scan request or a filter. <syntaxhighlight> <biocase:diagnostic severity="ERROR"> A SQL statement produced an error: SELECT unit.id, metadata.supplier_organisation, metadata.supplier_email, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) ORDER BY unit.id </biocase:diagnostic> </syntaxhighlight> That could have a number of reasons: One of the table/column names that were set up in the configuration does not exist (because it was renamed or removed) or the credentials used by the BPS do not have sufficient privileges. Simply copy the SQL statement and execute it manually on the database with a regular database client that will show you the detailed error message returned by the DBMS. <syntaxhighlight> <biocase:diagnostic severity="WARNING"> The mandatory child element SourceInstitutionID could not be created [0 existing, but 1 required]. The element /DataSets/DataSet/Units/Unit was also dropped. </biocase:diagnostic> <biocase:diagnostic severity="WARNING"> The mandatory child element Units could not be created [0 existing, but 1 required]. The element /DataSets/DataSet was also dropped. </biocase:diagnostic> </syntaxhighlight> One problem caused the next one in this sequence of warnings: The mandatory ABCD element ''SourceInstitutionID '' was either not mapped or empty; therefore the respective record (''unit'') was dropped. Because this happened for all units in the dataset, all units were dropped, resulting in an empty ''Units'' sub tree. The second warning tells you what happened then: Because the ''Units'' element is mandatory for the ''Dataset'' element, the whole dataset was dropped. As a result, the response ABCD document was empty. == Debugging a BioCASe Installation == In case the debug output of the web service doesn’t help you to find the problem, or if the BPS behaves not as intended without displaying any error messages, you should have a look at the debug logs written by the BPS. To turn on the Debug logs, go to the System Administration page and set <small>Debug logs</small> to <small>True</small>. [[File:instConfigTool.png|500px]] Redo the step that didn’t work and open the ''log'' folder of your BioCASe installation. There are several global log files (pywrapper_request.log, pywrapper_error.log, webapp_debug.log, webapp_error.log) and a debug files per data source (e.g. debug_flora.log). Open these files and see if they can give you a clue to the problem. If you still can’t get it running, [[Special:Contact|contact the BioCASe team]]. 718bc36029557a8132eae55289485765a0f4cd4b 0 40 282 94 2011-08-11T12:50:24Z JoergHoletschek 3 wikitext text/x-wiki This page shows a sample response ABCD2 document for a web service with 77 concepts mapped. For easier viewing, you can save the contents of the box into a file with the extension .xml and open that in Firefix or MS Internet Explorer (don't use Safari). You will then be able to collapse the different XML sub trees. <syntaxhighlight> <?xml version='1.0' encoding='UTF-8'?> <biocase:response xmlns:abcd="http://www.tdwg.org/schemas/abcd/2.06" xmlns:biocase="http://www.biocase.org/schemas/protocol/1.3" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.biocase.org/schemas/protocol/1.3 http://www.bgbm.org/biodivinf/schema/protocol_1_31.xsd"> <!-- XML generated by BioCASE PyWrapper software version 2.6.0. Made in Berlin. --> <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> <biocase:content recordCount="10" recordDropped="0" recordStart="0" totalSearchHits="11"> <abcd:DataSets> <abcd:DataSet> <abcd:TechnicalContacts> <abcd:TechnicalContact> <abcd:Name>Botanic Garden and Botanical Museum Berlin-Dahlem</abcd:Name> <abcd:Email>...@...</abcd:Email> <abcd:Address>Königin-Luise-Str. 6-8, D-14195 Berlin</abcd:Address> </abcd:TechnicalContact> </abcd:TechnicalContacts> <abcd:ContentContacts> <abcd:ContentContact> <abcd:Name>Dr. ...</abcd:Name> <abcd:Email>...@...</abcd:Email> <abcd:Phone>+49 ...</abcd:Phone> <abcd:Address>..., D-22609 Hamburg</abcd:Address> </abcd:ContentContact> </abcd:ContentContacts> <abcd:Metadata> <abcd:Description> <abcd:Representation abcd:language="EN"> <abcd:Title>Desmidiaceae Engels</abcd:Title> <abcd:Details>A new record of reference material of the Desmidiaceae of Germany was compiled, using the databank “Specify”. Aim of the project was to summarize and digitalize the data of the dried and otherwise preserved specimens stored in the German herbaria as well as of the living strains, cultured in culture collections. Many of the references are completed by figures of the labels or by scanned micrographs from preparations for the light- or scanning microscopes.</abcd:Details> <abcd:URI>http://www.gbif.de/botanik/projekte/desmidiaceae_hamburg_ffm/index_html</abcd:URI> </abcd:Representation> </abcd:Description> <abcd:IconURI>http://ww3.bgbm.org/providerResources/BzF.jpg</abcd:IconURI> <abcd:RevisionData> <abcd:Creators>Desmidiaceae Engels (BzF)</abcd:Creators> <abcd:DateModified>2008-01-23T00:00:00</abcd:DateModified> </abcd:RevisionData> <abcd:Owners> <abcd:Owner> <abcd:Organisation> <abcd:Name> <abcd:Representation abcd:language="EN"> <abcd:Text>Biozentrum Klein-Flottbek, Universität Hamburg</abcd:Text> <abcd:Abbreviation>BzF</abcd:Abbreviation> </abcd:Representation> </abcd:Name> </abcd:Organisation> </abcd:Owner> </abcd:Owners> <abcd:IPRStatements> <abcd:IPRDeclarations> <abcd:IPRDeclaration abcd:language="EN"> <abcd:Text>The Intellectual Property Rights are held by the legal owner or, in case of living persons, by the collector or determinator.</abcd:Text> </abcd:IPRDeclaration> </abcd:IPRDeclarations> <abcd:Copyrights> <abcd:Copyright abcd:language="EN"> <abcd:Text>No part of this data base may be copied or reproduced without written permission from the legal owner.</abcd:Text> </abcd:Copyright> </abcd:Copyrights> <abcd:TermsOfUseStatements> <abcd:TermsOfUse abcd:language="EN"> <abcd:Text>The use of the data is allowed only for non-profit scientific use and for non-profit nature conservation purpose. The database or part of it may only be used or copied by the written permission from the legal owner. If used for publication, we ask for a copy or an off-print.</abcd:Text> </abcd:TermsOfUse> </abcd:TermsOfUseStatements> <abcd:Disclaimers> <abcd:Disclaimer abcd:language="EN"> <abcd:Text>No responsibility is accepted for the accuracy of the information in this database.</abcd:Text> </abcd:Disclaimer> </abcd:Disclaimers> <abcd:Acknowledgements> <abcd:Acknowledgement abcd:language="EN"> <abcd:Text>The authors are much indebted to the Scientists of the Informatics Biodiversity Research Center der University of Kansas, USA (http://www.specifysoftware.org), who developed the ?Specify?-Software. Many thanks also to the curators of different Herbaria: Dr. Regine Jahn (BGBM, FU-Berlin), Dr. Dagmar Triebel (Botanische Staatssammlung München), Dr. Monika Steinhof (Übersee Museum Bremen), Christian Blümel (Universität Rostock) and Dr. Christian Printzen (Forschungsinstitut Senckenberg Frankfurt a.M.). Last not least thanks to the busy co-workers Sebastian Frerichmann, Maas Mollenhauer, Constanze Kovaci and Petra Kretschmann.</abcd:Text> </abcd:Acknowledgement> </abcd:Acknowledgements> <abcd:Citations> <abcd:Citation abcd:language="EN"> <abcd:Text>Engels, Monika 2003 - (continuously updated) Catalogus novus et amplificatus speciminum et viventium algarum Desmidiacearum (New and extended catalogue of herbarium specimen and living material of Desmidiaceae in Germany).</abcd:Text> </abcd:Citation> </abcd:Citations> </abcd:IPRStatements> </abcd:Metadata> <abcd:Units> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-6239</abcd:UnitID> <abcd:UnitReferences> <abcd:UnitReference> <abcd:TitleCitation>Algae aquae dulcis exsiccatae praecique scandinavicae, quas adjectis algis marinis chlorophyllaceis et phycochromaceis distribuerunt., Stockholm 1877-1889, Vol. 1-26</abcd:TitleCitation> </abcd:UnitReference> </abcd:UnitReferences> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Mesotaeniaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Zygnematales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Ancylonema nordenskiöldii</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:Identifiers> <abcd:Identifier> <abcd:PersonName> <abcd:FullName>Nordstedt, O.</abcd:FullName> </abcd:PersonName> </abcd:Identifier> </abcd:Identifiers> <abcd:Date> <abcd:DateText>1878-07-18</abcd:DateText> </abcd:Date> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>Specimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:MultiMediaObjects> <abcd:MultiMediaObject> <abcd:FileURI>http://ww2.bgbm.org/herbarium/Externe_Bildsammlungen/AlgenEngels/L-Scans-B-Konv/6239.jpg</abcd:FileURI> </abcd:MultiMediaObject> </abcd:MultiMediaObjects> <abcd:Gathering> <abcd:DateTime>1878<abcd:DateText>1878</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Nordstedt, O.</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Brixdal</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>NO</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Nordfjord</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:AreaDetail language="DE">see label</abcd:AreaDetail> <abcd:Biotope> <abcd:Name language="EN">other</abcd:Name> <abcd:Text language="DE">see label</abcd:Text> </abcd:Biotope> <abcd:Notes language="EN">Europe</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: B 4000 08107; Specimen Location: B.</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-3397</abcd:UnitID> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Actinotaenium cucurbita (Brébisson) Teiling ex Ruzicka et Pouzar 1978</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 108, tab. 17: 7a-e as Cosmarium cucurbita; taxon citation: Teiling, E. 1954: Actinotaenium genus Desmidiacearum resuscitatum. - Bot. Notiser 100, p. 406, fig. 66 (invalidy pblished). Validation: Ruzicka, J. &amp; Pouzar, Z. 1978:Erwägungen über die Taxonomie und Nomenklatur der Gattung Actinotaenium TEIL. - Folia Geobot. Phytotax., Praha, 13: p.44</abcd:Notes> </abcd:Identification> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Cosmarium cucurbita (Bréb.) Ralfs 1848</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>false</abcd:PreferredFlag> <abcd:Identifiers> <abcd:Identifier> <abcd:PersonName> <abcd:FullName>Siegmund, W.</abcd:FullName> </abcd:PersonName> </abcd:Identifier> </abcd:Identifiers> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Date> <abcd:DateText>1872-07</abcd:DateText> </abcd:Date> <abcd:Notes>p. 108, tab. 17: 7a-e</abcd:Notes> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>Specimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:MultiMediaObjects> <abcd:MultiMediaObject> <abcd:FileURI>http://ww2.bgbm.org/herbarium/Externe_Bildsammlungen/AlgenEngels/L-Scans-HBG-Konv/3397.jpg</abcd:FileURI> </abcd:MultiMediaObject> </abcd:MultiMediaObjects> <abcd:Gathering> <abcd:DateTime>1872<abcd:DateText>1872</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Siegmund, W.</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Reichenberg</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>DE</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Sachsen</abcd:AreaName> </abcd:NamedArea> <abcd:NamedArea> <abcd:AreaName>Dresden</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:AreaDetail language="DE">bei Reichenberg, see label</abcd:AreaDetail> <abcd:Biotope> <abcd:Name language="EN">other</abcd:Name> <abcd:Text language="DE">see label</abcd:Text> </abcd:Biotope> <abcd:Notes language="EN">Europe</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: 8186; Specimen Location: HBG.</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-2483</abcd:UnitID> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Cosmarium turgidum Bréb. 1848</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>false</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 110, tab. 32: 8</abcd:Notes> </abcd:Identification> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Actinotaenium turgidum (Brébisson) Teiling 1954 ex Ruzicka et Pouzar 1978</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>as Cosmarium turgidum p. 110, tab. 32: 8; taxon citation: invalidly published as A. turgidum (Bréb.) Teiling, E. 1954: Actinotaenium genus Desmidiacearum resuscitatum. - Bot. Notiser 100 - p.395, fig.21; Validation by R?ži?ka &amp; Pouzar 1978:Erwägungen über die Taxonomie und Nomenklatur der Gattung Actinotaenium TEIL. - Folia Geobot. Phytotax., Praha, 13: 13 - 66 p.60</abcd:Notes> </abcd:Identification> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Pleurotaenium turgidum De Bary 1858</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>false</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>De Bary, A: Untersuchungen über die Familie der Conjugaten (Zygnemeen und Desmidieen)., Leipzig 1858</abcd:TitleCitation> <abcd:CitationDetail>p. 91</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 75, tab. 5: 31; invalidly published</abcd:Notes> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>Specimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:Gathering> <abcd:DateTime>0<abcd:DateText>0</abcd:DateText> </abcd:DateTime> <abcd:LocalityText language="DE">Neukirch</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>DE</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Sachsen</abcd:AreaName> </abcd:NamedArea> <abcd:NamedArea> <abcd:AreaName>Chemnitz</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:AreaDetail language="DE">Oberlausitz, Neukirch</abcd:AreaDetail> <abcd:Biotope> <abcd:Name language="EN" /> <abcd:Text language="DE" /> </abcd:Biotope> <abcd:Notes language="EN">Europe</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: 7474; Specimen Location: HBG. together with Closterium hybridum, Pl. truncatum</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-4603</abcd:UnitID> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Arthrodesmus octocornis var. major Roy &amp; Biss. 1894</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:Identifiers> <abcd:Identifier> <abcd:PersonName> <abcd:FullName>Richter, Paul</abcd:FullName> </abcd:PersonName> </abcd:Identifier> </abcd:Identifiers> <abcd:Notes>see label</abcd:Notes> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>Specimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:MultiMediaObjects> <abcd:MultiMediaObject> <abcd:FileURI>http://ww2.bgbm.org/herbarium/Externe_Bildsammlungen/AlgenEngels/L-Scans-HBG-Konv/4602.jpg</abcd:FileURI> </abcd:MultiMediaObject> </abcd:MultiMediaObjects> <abcd:Gathering> <abcd:DateTime>0<abcd:DateText>0</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Richter, Paul</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Leipzig</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>DE</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Sachsen</abcd:AreaName> </abcd:NamedArea> <abcd:NamedArea> <abcd:AreaName>Leipzig</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:AreaDetail language="DE" /> <abcd:Biotope> <abcd:Name language="EN" /> <abcd:Text language="DE" /> </abcd:Biotope> <abcd:Notes language="EN">Europe</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: 8773; Specimen Location: HBG.</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-6764</abcd:UnitID> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Actinotaenium curtum var. attenuatum (Bréb.) Teil. ex Ruzicka &amp; Pouzar 1978</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>as Cosmarium attenuatum p. 110, tab. 17: 9; taxon citation: Teiling, E., 1954: Actinotaenium genus Desmidiacearum resuscitatum. - Bot. Not., Lund, 1954, p. 391, fig. 2; invaldly published; Validation: Ruzicka, J., &amp; Pouzar, Z., 1978: Erwägungen über die Taxonomie und Nomenklatur der Gattung ActinotaeniumTEIL. - Folia geobot. Phytotax., Praha, 13: 33-66., p.47</abcd:Notes> </abcd:Identification> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Cosmarium attenuatum Ralfs 1848</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>false</abcd:PreferredFlag> <abcd:Identifiers> <abcd:Identifier> <abcd:PersonName> <abcd:FullName>Richter, Paul</abcd:FullName> </abcd:PersonName> </abcd:Identifier> </abcd:Identifiers> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 110, tab. 17: 9a-d</abcd:Notes> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>Specimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:Gathering> <abcd:DateTime>1898<abcd:DateText>1898</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Rogers,</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Ennepetal-Milspe</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>DE</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Nordrhein-Westfalen</abcd:AreaName> </abcd:NamedArea> <abcd:NamedArea> <abcd:AreaName>Hagen</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:AreaDetail language="DE" /> <abcd:Biotope> <abcd:Name language="EN" /> <abcd:Text language="DE">an einer Felsenwand</abcd:Text> </abcd:Biotope> <abcd:Notes language="EN">Europe</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: B 4000 08307; Specimen Location: B.</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-3698</abcd:UnitID> <abcd:UnitReferences> <abcd:UnitReference> <abcd:TitleCitation>Algae aquae dulcis exsiccatae praecique scandinavicae, quas adjectis algis marinis chlorophyllaceis et phycochromaceis distribuerunt., Lund 1896-1903, Vol. 27-34</abcd:TitleCitation> </abcd:UnitReference> </abcd:UnitReferences> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Actinotaenium palangula (Brébisson) Teiling 1954 ex Ruzicka et Pouzar 1978</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 212; original figure: Brébisson 1856: p.132, tab.1: 21; taxon citation: Ruzicka, J. &amp; Pouzar, Z., 1978: Erwägungen über die Taxonomie und Nomenklatur der Gattung Actinotaenium TEIL. - Folia Geobot. Phytotax., Praha, 13: p.46</abcd:Notes> </abcd:Identification> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Cosmarium palangula Brébisson in Ralfs 1848</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>false</abcd:PreferredFlag> <abcd:Identifiers> <abcd:Identifier> <abcd:PersonName> <abcd:FullName>Bohlin, Knut</abcd:FullName> </abcd:PersonName> </abcd:Identifier> </abcd:Identifiers> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Date> <abcd:DateText>1898-07-06</abcd:DateText> </abcd:Date> <abcd:Notes>p. 212; original figure: Brébisson 1856: p.132, tab.1: 21</abcd:Notes> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>Specimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:MultiMediaObjects> <abcd:MultiMediaObject> <abcd:FileURI>http://ww2.bgbm.org/herbarium/Externe_Bildsammlungen/AlgenEngels/L-Scans-HBG-Konv/3690-3703.jpg</abcd:FileURI> </abcd:MultiMediaObject> </abcd:MultiMediaObjects> <abcd:Gathering> <abcd:DateTime>1898<abcd:DateText>1898</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Bohlin, Knut</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Pico da Cruz</abcd:LocalityText> <abcd:AreaDetail language="DE" /> <abcd:Biotope> <abcd:Name language="EN" /> <abcd:Text language="DE" /> </abcd:Biotope> <abcd:Notes language="EN">Africa</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: 8328; Specimen Location: HBG. sample contains: Cosmarium ralfsii var. azoricum, Cosmarium palangula, Cosmarium pyramidatum, Cosmarium venustum, Euastru didelta f. scorbiculata, Staurastrum muticum var. depressum, Staurastrum scabrum</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-4569</abcd:UnitID> <abcd:UnitReferences> <abcd:UnitReference> <abcd:TitleCitation>Die Algen Europa's (Fortsetzung der Algen Sachsens, resp. Mittel-Europa's)., Dresden 1861 - 1879, Vol. Dec. 101 - 259</abcd:TitleCitation> </abcd:UnitReference> </abcd:UnitReferences> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Staurodesmus convergens (Ehrenberg ex Ralfs) Teiling 1967</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 118, tab. 20: 3a-d; taxon citation: Teiling, E., 1948: Staurodesmus, genus novum.Containing monospinous desmids. - Bot. Not., Lund, 1948: p. 57</abcd:Notes> </abcd:Identification> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Arthrodesmus convergens Ehrenberg ex Ralfs 1848</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>false</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 118, tab. 20: 3a-d</abcd:Notes> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>Specimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:Gathering> <abcd:DateTime>0<abcd:DateText>0</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Rostock, M.</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Dretschen</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>DE</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Sachsen</abcd:AreaName> </abcd:NamedArea> <abcd:NamedArea> <abcd:AreaName>Bautzen</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:AreaDetail language="DE" /> <abcd:Biotope> <abcd:Name language="EN">peat bog</abcd:Name> <abcd:Text language="DE" /> </abcd:Biotope> <abcd:Notes language="EN">Europe</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: 8751; Specimen Location: HBG.</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-7071</abcd:UnitID> <abcd:UnitReferences> <abcd:UnitReference> <abcd:TitleCitation>Kryptogamae exsiccatae editae a museo hist. natur. vindobonensi., Wien 1894 ff</abcd:TitleCitation> </abcd:UnitReference> </abcd:UnitReferences> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Arthrodesmus quadricaudatus</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:Identifiers> <abcd:Identifier> <abcd:PersonName> <abcd:FullName>Hansgirg, A.</abcd:FullName> </abcd:PersonName> </abcd:Identifier> </abcd:Identifiers> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>Specimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:Gathering> <abcd:DateTime>0<abcd:DateText>0</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Hansgirg, A.</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Bombay</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>IN</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Bombay</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:AreaDetail language="DE" /> <abcd:Biotope> <abcd:Name language="EN">other</abcd:Name> <abcd:Text language="DE">see label</abcd:Text> </abcd:Biotope> <abcd:Notes language="EN">Asia</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: B 4000 08385; Specimen Location: B. sample contains: Cosmarium nitidulum, Arthrodesmus quadricaudatus</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-6250</abcd:UnitID> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Staurodesmus incus (Brébisson ex Ralfs) Teiling 1967</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 118, tab. 20: 4 as Arthrodesmus incus; taxon citation: Teiling, E., 1967: The desmid genus Staurodesmus. A taxonomic study. Ark. f. Bot. II, 6(11), p. p. 511, tab. 5: 9</abcd:Notes> </abcd:Identification> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Arthrodesmus incus (Bréb.) Hassall ex Ralfs 1848</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>false</abcd:PreferredFlag> <abcd:Identifiers> <abcd:Identifier> <abcd:PersonName> <abcd:FullName>Borge, O.</abcd:FullName> </abcd:PersonName> </abcd:Identifier> </abcd:Identifiers> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Date> <abcd:DateText>1889-08-08</abcd:DateText> </abcd:Date> <abcd:Notes>p. 118, tab. 20: 4a-l</abcd:Notes> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>Specimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:Gathering> <abcd:DateTime>1889<abcd:DateText>1889</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Borge, O.</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Väddö</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>SE</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Upland</abcd:AreaName> </abcd:NamedArea> <abcd:NamedArea> <abcd:AreaName>Väddö</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:AreaDetail language="DE" /> <abcd:Biotope> <abcd:Name language="EN" /> <abcd:Text language="DE" /> </abcd:Biotope> <abcd:Notes language="EN">Europe</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: B 4000 32088; Specimen Location: B.</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-6897</abcd:UnitID> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Actinotaenium cucurbita (Brébisson) Teiling ex Ruzicka et Pouzar 1978</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 108, tab. 17: 7a-e as Cosmarium cucurbita; taxon citation: Teiling, E. 1954: Actinotaenium genus Desmidiacearum resuscitatum. - Bot. Notiser 100, p. 406, fig. 66 (invalidy pblished). Validation: Ruzicka, J. &amp; Pouzar, Z. 1978:Erwägungen über die Taxonomie und Nomenklatur der Gattung Actinotaenium TEIL. - Folia Geobot. Phytotax., Praha, 13: p.44</abcd:Notes> </abcd:Identification> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Cosmarium cucurbita (Bréb.) Ralfs 1848</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>false</abcd:PreferredFlag> <abcd:Identifiers> <abcd:Identifier> <abcd:PersonName> <abcd:FullName>Bulnheim, O.</abcd:FullName> </abcd:PersonName> </abcd:Identifier> </abcd:Identifiers> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Date> <abcd:DateText>1883-07</abcd:DateText> </abcd:Date> <abcd:Notes>p. 108, tab. 17: 7a-e</abcd:Notes> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>Specimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:MultiMediaObjects> <abcd:MultiMediaObject> <abcd:FileURI>http://ww2.bgbm.org/herbarium/Externe_Bildsammlungen/AlgenEngels/L-Scans-B-Konv/6897-6899.jpg</abcd:FileURI> </abcd:MultiMediaObject> </abcd:MultiMediaObjects> <abcd:Gathering> <abcd:DateTime>1883<abcd:DateText>1883</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Bulnheim, O.</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Leipzig</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>DE</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Sachsen</abcd:AreaName> </abcd:NamedArea> <abcd:NamedArea> <abcd:AreaName>Leipzig</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:AreaDetail language="DE">see label</abcd:AreaDetail> <abcd:Biotope> <abcd:Name language="EN" /> <abcd:Text language="DE" /> </abcd:Biotope> <abcd:Notes language="EN">Europe</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: B 4000 08358; Specimen Location: B. sample contains: Cosmarium cucurbita, Euastrum didelta, Closterium intermedium</abcd:Notes> </abcd:Unit> </abcd:Units> </abcd:DataSet> </abcd:DataSets> </biocase:content> <biocase:diagnostics> <biocase:diagnostic severity="INFO">Datasource wrapper AlgenEngels requested</biocase:diagnostic> <biocase:diagnostic severity="INFO">Reading PSF from C:\Workspace\bps2\config\datasources\AlgenEngels\provider_setup_file.xml</biocase:diagnostic> <biocase:diagnostic severity="DEBUG">PSF: PSF=C:\Workspace\bps2\config\datasources\AlgenEngels\provider_setup_file.xml, recLimit=0, loglevel=40, user=WebUser, database=AlgenEngels, dbIP=, dbms=odbc_tsql, encoding=latin_1, schemas={u'http://www.tdwg.org/schemas/abcd/2.06': &lt;biocase.wrapper.psf_handler.SupportedSchema instance at 0x00EB1E90&gt;}, tablegraph=GRAPH: graph: meta, determination-taxonNames, higherTaxon-taxonNames, images-specimen, namedAreas-collectingEvent, siteFeatures-collectingEvent, specimen-determination, specimen-collectingEvent, unitReference-specimen, +++ ALIAS2TABLE: {u'collectingEvent': u'abcd_CollectingEvent', u'taxonNames': u'abcd_TaxonName', u'higherTaxon': u'abcd_HigherTaxa', u'specimen': u'abcd_Object', u'siteFeatures': u'abcd_SiteFeatureTerms', u'meta': u'metadata', u'determination': u'abcd_Determination', u'images': u'abcd_Images', u'unitReference': u'abcd_UnitReference', u'namedAreas': u'abcd_NamedAreaNames'}</biocase:diagnostic> </biocase:diagnostics> </biocase:response> </syntaxhighlight> c2f6ba0ef320b5182885b6c363eb1d32aa00340b 2 73 284 2011-08-12T11:18:04Z DavidFichtmueller 4 Created page with " addOnloadHook(function(){ addPortletLink("p-personal","http://wiki.bgbm.org/bps/index.php?title=User:DavidFichtmueller/monobook.js&action=edit","monobook.js", "t-edit-vector","..." javascript text/javascript addOnloadHook(function(){ addPortletLink("p-personal","http://wiki.bgbm.org/bps/index.php?title=User:DavidFichtmueller/monobook.js&action=edit","monobook.js", "t-edit-vector","edit vector.js","v",document.getElementById("pt-preferences")); var mainNavigation = document.getElementById("t-specialpages").parentNode; var specialpages= document.getElementById("t-specialpages") addPortletLink("p-tb","http://wiki.bgbm.org/bps/index.php/Special:RecentChanges","Recent Changes", "t-recent-changes","Recent Changes","s",specialpages); if(wgNamespaceNumber==2 || wgNamespaceNumber==3){ //user namespace or user discussion namespace var slashPos = wgTitle.indexOf('/'); var username = wgTitle; if(slashPos != -1){ // it's a subpage username = username.substring(0,slashPos); } addPortletLink("p-namespaces","http://wiki.bgbm.org/bps/index.php/Special:Contributions/"+username,"Contributions", "ca-contributions","Contributions of "+username,"b"); } } ); //closes addOnloadHook( ec7ba17cc28a971a7fa81eb67177ac7e4fc54e90 285 284 2011-08-12T11:21:42Z DavidFichtmueller 4 javascript text/javascript addOnloadHook(function(){ addPortletLink("p-personal","http://wiki.bgbm.org/bps/index.php?title=User:DavidFichtmueller/monobook.js&action=edit","monobook.js", "t-edit-vector","edit vector.js","v",document.getElementById("pt-preferences")); var mainNavigation = document.getElementById("t-whatlinkshere").parentNode; var specialpages= document.getElementById("t-specialpages") addPortletLink("p-tb","http://wiki.bgbm.org/bps/index.php/Special:RecentChanges","Recent Changes", "t-recent-changes","Recent Changes","s",specialpages); if(wgNamespaceNumber==2 || wgNamespaceNumber==3){ //user namespace or user discussion namespace var slashPos = wgTitle.indexOf('/'); var username = wgTitle; if(slashPos != -1){ // it's a subpage username = username.substring(0,slashPos); } addPortletLink("p-namespaces","http://wiki.bgbm.org/bps/index.php/Special:Contributions/"+username,"Contributions", "ca-contributions","Contributions of "+username,"b"); } } ); //closes addOnloadHook( a37e6c3e80a5b66ecf2226854610bec2d11ef417 286 285 2011-08-12T11:22:21Z DavidFichtmueller 4 javascript text/javascript addOnloadHook(function(){ addPortletLink("p-personal","http://wiki.bgbm.org/bps/index.php?title=User:DavidFichtmueller/monobook.js&action=edit","monobook.js", "t-edit-vector","edit vector.js","v",document.getElementById("pt-preferences")); var mainNavigation = document.getElementById("t-whatlinkshere").parentNode; var whatlinkshere= document.getElementById("t-whatlinkshere") addPortletLink("p-tb","http://wiki.bgbm.org/bps/index.php/Special:RecentChanges","Recent Changes", "t-recent-changes","Recent Changes","s",whatlinkshere); if(wgNamespaceNumber==2 || wgNamespaceNumber==3){ //user namespace or user discussion namespace var slashPos = wgTitle.indexOf('/'); var username = wgTitle; if(slashPos != -1){ // it's a subpage username = username.substring(0,slashPos); } addPortletLink("p-namespaces","http://wiki.bgbm.org/bps/index.php/Special:Contributions/"+username,"Contributions", "ca-contributions","Contributions of "+username,"b"); } } ); //closes addOnloadHook( 10235e5f25e563cb0ef60d4a9b5cd43c4ccec7e1 287 286 2011-08-12T11:27:42Z DavidFichtmueller 4 javascript text/javascript addOnloadHook(function(){ addPortletLink("p-personal","http://wiki.bgbm.org/bps/index.php?title=User:DavidFichtmueller/monobook.js&action=edit","monobook.js", "t-edit-vector","edit vector.js","v",document.getElementById("pt-preferences")); var mainNavigation = document.getElementById("t-whatlinkshere").parentNode; var whatlinkshere= document.getElementById("t-whatlinkshere") addPortletLink("p-tb","http://wiki.bgbm.org/bps/index.php/Special:RecentChanges","Recent Changes", "t-recent-changes","Recent Changes","s",whatlinkshere); if(wgNamespaceNumber==2 || wgNamespaceNumber==3){ //user namespace or user discussion namespace var slashPos = wgTitle.indexOf('/'); var username = wgTitle; if(slashPos != -1){ // it's a subpage username = username.substring(0,slashPos); } addPortletLink("p-cationc","http://wiki.bgbm.org/bps/index.php/Special:Contributions/"+username,"Contributions", "ca-contributions","Contributions of "+username,"b",document.getElementById("ca-talk").nextElementSibling); document.getElementById("ca-talk").style.marginRight="1em"; document.getElementById("ca-contributions").style.marginRight="1.6em"; } } ); //closes addOnloadHook( 0cfa29fcab9688286c86e40fd34bbddd115ad1b5 288 287 2011-08-12T11:29:56Z DavidFichtmueller 4 javascript text/javascript addOnloadHook(function(){ addPortletLink("p-personal","http://wiki.bgbm.org/bps/index.php?title=User:DavidFichtmueller/monobook.js&action=edit","monobook.js", "t-edit-vector","edit vector.js","v",document.getElementById("pt-preferences")); var mainNavigation = document.getElementById("t-whatlinkshere").parentNode; var whatlinkshere= document.getElementById("t-whatlinkshere") addPortletLink("p-tb","http://wiki.bgbm.org/bps/index.php/Special:RecentChanges","Recent Changes", "t-recent-changes","Recent Changes","s",whatlinkshere); if(wgNamespaceNumber==2 || wgNamespaceNumber==3){ //user namespace or user discussion namespace var slashPos = wgTitle.indexOf('/'); var username = wgTitle; if(slashPos != -1){ // it's a subpage username = username.substring(0,slashPos); } addPortletLink("p-cactions","http://wiki.bgbm.org/bps/index.php/Special:Contributions/"+username,"Contributions", "ca-contributions","Contributions of "+username,"b",document.getElementById("ca-talk").nextElementSibling); document.getElementById("ca-talk").style.marginRight="0.3em"; document.getElementById("ca-contributions").style.marginRight="1.6em"; } } ); //closes addOnloadHook( 6c5e8da39f218fc0bd0b03b8edd3b7a960ef1b43 289 288 2011-08-12T11:32:14Z DavidFichtmueller 4 javascript text/javascript addOnloadHook(function(){ addPortletLink("p-personal","http://wiki.bgbm.org/bps/index.php?title=User:DavidFichtmueller/monobook.js&action=edit","monobook.js", "t-edit-vector","edit vector.js","v",document.getElementById("pt-preferences")); var firstElement = document.getElementById("t-specialpages").firstElementChild; addPortletLink("p-tb","http://wiki.bgbm.org/bps/index.php/Special:RecentChanges","Recent Changes", "t-recent-changes","Recent Changes","s",firstElement); if(wgNamespaceNumber==2 || wgNamespaceNumber==3){ //user namespace or user discussion namespace var slashPos = wgTitle.indexOf('/'); var username = wgTitle; if(slashPos != -1){ // it's a subpage username = username.substring(0,slashPos); } addPortletLink("p-cactions","http://wiki.bgbm.org/bps/index.php/Special:Contributions/"+username,"Contributions", "ca-contributions","Contributions of "+username,"b",document.getElementById("ca-talk").nextElementSibling); document.getElementById("ca-talk").style.marginRight="0.3em"; document.getElementById("ca-contributions").style.marginRight="1.6em"; } } ); //closes addOnloadHook( ce6213161b104a93dbb4c91018657251510df5e4 290 289 2011-08-12T11:33:11Z DavidFichtmueller 4 javascript text/javascript addOnloadHook(function(){ addPortletLink("p-personal","http://wiki.bgbm.org/bps/index.php?title=User:DavidFichtmueller/monobook.js&action=edit","monobook.js", "t-edit-vector","edit vector.js","v",document.getElementById("pt-preferences")); var firstElement = document.getElementById("t-specialpages").parentNode.firstElementChild; addPortletLink("p-tb","http://wiki.bgbm.org/bps/index.php/Special:RecentChanges","Recent Changes", "t-recent-changes","Recent Changes","s",firstElement); if(wgNamespaceNumber==2 || wgNamespaceNumber==3){ //user namespace or user discussion namespace var slashPos = wgTitle.indexOf('/'); var username = wgTitle; if(slashPos != -1){ // it's a subpage username = username.substring(0,slashPos); } addPortletLink("p-cactions","http://wiki.bgbm.org/bps/index.php/Special:Contributions/"+username,"Contributions", "ca-contributions","Contributions of "+username,"b",document.getElementById("ca-talk").nextElementSibling); document.getElementById("ca-talk").style.marginRight="0.3em"; document.getElementById("ca-contributions").style.marginRight="1.6em"; } } ); //closes addOnloadHook( 150752909d095bf2d9a082fc84bf81a0ee1bfe29 0 23 297 260 2011-08-15T12:31:54Z JoergHoletschek 3 /* Version 2.6.1 [upcoming] */ wikitext text/x-wiki == Version 2 History == ===Version 2.6.1 [upcoming]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.5.5 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. See example [http://ww3.bgbm.org/bps2/ScreenShots?action=AttachFile&do=get&target=config_tool4.gif here] * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool be4f70a46a94e05765aa8226b339accbee582554 0 49 298 251 2011-08-16T10:25:09Z JoergHoletschek 3 /* What is the BioCASe Provider Software? */ wikitext text/x-wiki == What is the BioCASe Provider Software? == The BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. The Provider Software is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. The Provider Software is a suite of tools that need to be installed on the data provider’s web server. The core component is the PyWrapper, an XML/CGI database interface written in the Python scripting language that allows a standardized access to a variety of database management systems and arbitrarily structured databases. Grouped around this are a number of tools for configuring, testing and debugging a BioCASe installation or web service. Even though BioCASe can be used for any conceptual XML data schema, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the [http://www.biocase.org BioCASe network] and the [http://www.gbif.org Global Biodiversity Information Facility]. == Requirements for Using the BPS == Before using the BioCASe Provider Software, two requirements need to be checked: * For one thing, the data to be published needs to be stored in an SQL compliant relational database management system (DBMS), with the data model being well documented or at least knowledge about the data model existing on someone’s head. Non-SQL DBMS are not supported. Currently, the BioCASe Provider Software can connect to the following databases: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * For installing BioCASe, you need to have a web server (Apache or Microsoft IIS) that is permanently connected to the Internet. This can be an institutional server of a hosting service (even though it’s usually a pain to get privileges to install software for the latter). Dial-up connections won’t work. == Steps == Once you’ve made sure the BioCASe Provider Software is the right choice for you ([[Special:Contact|contact the BioCASe team]] if you feel unsafe with your decision), you need to do the following steps: # Think about which information you want to publish to the network. This involves settling the question of ownership of the data and the terms of use, license and usage restrictions for the published data. Also you should consider which information should '''not''' be published because they might affect endangered species (options are to block certain occurrences from publication or to blur the exact locality information). # Prepare the database for publication as described in the [[Preparation]] tutorial. This includes the decision on whether to publish the live database or a snapshot, the creation of a table for metadata and, in certain cases, some data transformation. # Follow the [[Installation]] Tutorial to install the BioCASe Provider Software. # Set up a BioCASe Datasource and connect it to your database, then configure the database structure as described in the [[DatasourceSetup]] tutorial. # Follow the [[ABCD2Mapping]] tutorial to create a mapping for the schema you want to use. # Test and debug your BioCASe web service. Read the [[Debugging]] tutorial to learn how to do this. # Once you’re done with setting up and configuring the BioCASe Provider Software, you can register the BioCASe web service with the information network you want to publish your data to, for example with [http://www.gbif.org GBIF]. We strongly recommend to [[Special:Contact|contact the BioCASe team]] for checking your web service first as we might be able to foresee problems that are more difficult to resolve after registration. == Support == In case you run into difficulties during the installation and setup, you should * have a look at the [[FAQ]] which might list the solution for your question, * ask other BioCASe data providers for help, * get in contact with the [[Special:Contact|BioCASe team]]. Please describe exactly you environment (operating system, DBMS used), the steps that cause the problem and any error messages and hints you get. Screenshot are a big plus here. a99971b835a119818e3ecff6deec4f2d12494c98 299 298 2011-08-16T10:26:39Z JoergHoletschek 3 /* Requirements for Using the BPS */ wikitext text/x-wiki == What is the BioCASe Provider Software? == The BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. The Provider Software is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. The Provider Software is a suite of tools that need to be installed on the data provider’s web server. The core component is the PyWrapper, an XML/CGI database interface written in the Python scripting language that allows a standardized access to a variety of database management systems and arbitrarily structured databases. Grouped around this are a number of tools for configuring, testing and debugging a BioCASe installation or web service. Even though BioCASe can be used for any conceptual XML data schema, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the [http://www.biocase.org BioCASe network] and the [http://www.gbif.org Global Biodiversity Information Facility]. == Requirements for Using the BPS == Before using the BioCASe Provider Software, two requirements need to be checked: * For one thing, the data to be published needs to be stored in an SQL compliant relational database management system (DBMS), with the data model being well documented or at least knowledge about the data model existing on someone’s head. Non-SQL DBMS are not supported. Currently, the BioCASe Provider Software can connect to the following databases: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * For installing BioCASe, you need to have a web server (Apache or Microsoft IIS) that is permanently connected to the Internet. This can be an institutional server or a hosting service (even though it’s usually a pain to get privileges to install software for the latter). Dial-up connections won’t work. == Steps == Once you’ve made sure the BioCASe Provider Software is the right choice for you ([[Special:Contact|contact the BioCASe team]] if you feel unsafe with your decision), you need to do the following steps: # Think about which information you want to publish to the network. This involves settling the question of ownership of the data and the terms of use, license and usage restrictions for the published data. Also you should consider which information should '''not''' be published because they might affect endangered species (options are to block certain occurrences from publication or to blur the exact locality information). # Prepare the database for publication as described in the [[Preparation]] tutorial. This includes the decision on whether to publish the live database or a snapshot, the creation of a table for metadata and, in certain cases, some data transformation. # Follow the [[Installation]] Tutorial to install the BioCASe Provider Software. # Set up a BioCASe Datasource and connect it to your database, then configure the database structure as described in the [[DatasourceSetup]] tutorial. # Follow the [[ABCD2Mapping]] tutorial to create a mapping for the schema you want to use. # Test and debug your BioCASe web service. Read the [[Debugging]] tutorial to learn how to do this. # Once you’re done with setting up and configuring the BioCASe Provider Software, you can register the BioCASe web service with the information network you want to publish your data to, for example with [http://www.gbif.org GBIF]. We strongly recommend to [[Special:Contact|contact the BioCASe team]] for checking your web service first as we might be able to foresee problems that are more difficult to resolve after registration. == Support == In case you run into difficulties during the installation and setup, you should * have a look at the [[FAQ]] which might list the solution for your question, * ask other BioCASe data providers for help, * get in contact with the [[Special:Contact|BioCASe team]]. Please describe exactly you environment (operating system, DBMS used), the steps that cause the problem and any error messages and hints you get. Screenshot are a big plus here. c1d79a7aaf98ccbb4101e663154ed35d558e2655 302 299 2011-08-16T13:10:51Z JoergHoletschek 3 /* What is the BioCASe Provider Software? */ wikitext text/x-wiki == What is the BioCASe Provider Software? == The BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. The Provider Software is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. The Provider Software is a suite of tools that need to be installed on the data provider’s web server. The core component is the PyWrapper, an XML/CGI database interface written in Python that allows a standardized access to a variety of database management systems and arbitrarily structured databases. Grouped around this are a number of tools for configuring, testing and debugging a BioCASe installation or web service. Even though BioCASe can be used for any conceptual XML data schema, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the [http://www.biocase.org BioCASe network] and the [http://www.gbif.org Global Biodiversity Information Facility]. == Requirements for Using the BPS == Before using the BioCASe Provider Software, two requirements need to be checked: * For one thing, the data to be published needs to be stored in an SQL compliant relational database management system (DBMS), with the data model being well documented or at least knowledge about the data model existing on someone’s head. Non-SQL DBMS are not supported. Currently, the BioCASe Provider Software can connect to the following databases: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * For installing BioCASe, you need to have a web server (Apache or Microsoft IIS) that is permanently connected to the Internet. This can be an institutional server or a hosting service (even though it’s usually a pain to get privileges to install software for the latter). Dial-up connections won’t work. == Steps == Once you’ve made sure the BioCASe Provider Software is the right choice for you ([[Special:Contact|contact the BioCASe team]] if you feel unsafe with your decision), you need to do the following steps: # Think about which information you want to publish to the network. This involves settling the question of ownership of the data and the terms of use, license and usage restrictions for the published data. Also you should consider which information should '''not''' be published because they might affect endangered species (options are to block certain occurrences from publication or to blur the exact locality information). # Prepare the database for publication as described in the [[Preparation]] tutorial. This includes the decision on whether to publish the live database or a snapshot, the creation of a table for metadata and, in certain cases, some data transformation. # Follow the [[Installation]] Tutorial to install the BioCASe Provider Software. # Set up a BioCASe Datasource and connect it to your database, then configure the database structure as described in the [[DatasourceSetup]] tutorial. # Follow the [[ABCD2Mapping]] tutorial to create a mapping for the schema you want to use. # Test and debug your BioCASe web service. Read the [[Debugging]] tutorial to learn how to do this. # Once you’re done with setting up and configuring the BioCASe Provider Software, you can register the BioCASe web service with the information network you want to publish your data to, for example with [http://www.gbif.org GBIF]. We strongly recommend to [[Special:Contact|contact the BioCASe team]] for checking your web service first as we might be able to foresee problems that are more difficult to resolve after registration. == Support == In case you run into difficulties during the installation and setup, you should * have a look at the [[FAQ]] which might list the solution for your question, * ask other BioCASe data providers for help, * get in contact with the [[Special:Contact|BioCASe team]]. Please describe exactly you environment (operating system, DBMS used), the steps that cause the problem and any error messages and hints you get. Screenshot are a big plus here. 8107813079ac9297cb41f18f4161b472485b2536 303 302 2011-08-16T13:17:30Z JoergHoletschek 3 /* Requirements for Using the BPS */ wikitext text/x-wiki == What is the BioCASe Provider Software? == The BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. The Provider Software is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. The Provider Software is a suite of tools that need to be installed on the data provider’s web server. The core component is the PyWrapper, an XML/CGI database interface written in Python that allows a standardized access to a variety of database management systems and arbitrarily structured databases. Grouped around this are a number of tools for configuring, testing and debugging a BioCASe installation or web service. Even though BioCASe can be used for any conceptual XML data schema, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the [http://www.biocase.org BioCASe network] and the [http://www.gbif.org Global Biodiversity Information Facility]. == Requirements for Using the BPS == Before using the BioCASe Provider Software, three requirements need to be checked: * For one thing, the data to be published needs to be stored in an SQL compliant relational database management system (DBMS), with the data model being well documented or at least knowledge about the data model existing on someone’s head. Non-SQL DBMS are not supported. Currently, the BioCASe Provider Software can connect to the following databases: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * For installing BioCASe, you need to have a web server (Apache or Microsoft IIS) that is permanently connected to the Internet. This can be an institutional server or a hosting service (even though it’s usually a pain to get privileges to install software for the latter). Dial-up connections won’t work. * For the web server in question, you must have the possibility to install Python (if it is not installed yet) and Python packages. Even though this can be done by the IT staff, you should make sure that there are no institutional policies colliding. == Steps == Once you’ve made sure the BioCASe Provider Software is the right choice for you ([[Special:Contact|contact the BioCASe team]] if you feel unsafe with your decision), you need to do the following steps: # Think about which information you want to publish to the network. This involves settling the question of ownership of the data and the terms of use, license and usage restrictions for the published data. Also you should consider which information should '''not''' be published because they might affect endangered species (options are to block certain occurrences from publication or to blur the exact locality information). # Prepare the database for publication as described in the [[Preparation]] tutorial. This includes the decision on whether to publish the live database or a snapshot, the creation of a table for metadata and, in certain cases, some data transformation. # Follow the [[Installation]] Tutorial to install the BioCASe Provider Software. # Set up a BioCASe Datasource and connect it to your database, then configure the database structure as described in the [[DatasourceSetup]] tutorial. # Follow the [[ABCD2Mapping]] tutorial to create a mapping for the schema you want to use. # Test and debug your BioCASe web service. Read the [[Debugging]] tutorial to learn how to do this. # Once you’re done with setting up and configuring the BioCASe Provider Software, you can register the BioCASe web service with the information network you want to publish your data to, for example with [http://www.gbif.org GBIF]. We strongly recommend to [[Special:Contact|contact the BioCASe team]] for checking your web service first as we might be able to foresee problems that are more difficult to resolve after registration. == Support == In case you run into difficulties during the installation and setup, you should * have a look at the [[FAQ]] which might list the solution for your question, * ask other BioCASe data providers for help, * get in contact with the [[Special:Contact|BioCASe team]]. Please describe exactly you environment (operating system, DBMS used), the steps that cause the problem and any error messages and hints you get. Screenshot are a big plus here. e3b2ca7c93155c3a8e8f452167008264b1f63bfa 0 6 300 250 2011-08-16T10:35:27Z JoergHoletschek 3 /* Running the Setup Script */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given at the beginning out the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] All of the questions do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Import configuration from existing installation:</strong> Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.foobar.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.foobar.org/biocase'', for example. After answering these three questions, the script will update your files. At the end of the output it will print the lines you will need later to create the link, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so will compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.foobar.org/biocase''. If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can a pain; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the DBMS used by the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to he library test page that shows the status of installed libraries. On a vanilla BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries ship with the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' in the browser window), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown [[Installation#Linking_BioCASe_and_Your_Web_Server|here]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] 7cbf5318a5908ea03319f1ae33c137bd1e06a14e 301 300 2011-08-16T10:36:07Z JoergHoletschek 3 /* Running the Setup Script */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given at the beginning of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] All of the questions do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Import configuration from existing installation:</strong> Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.foobar.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.foobar.org/biocase'', for example. After answering these three questions, the script will update your files. At the end of the output it will print the lines you will need later to create the link, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so will compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.foobar.org/biocase''. If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can a pain; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the DBMS used by the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to he library test page that shows the status of installed libraries. On a vanilla BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries ship with the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' in the browser window), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown [[Installation#Linking_BioCASe_and_Your_Web_Server|here]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] 069336ef98259e279b3ba4364703ddd4d03c338c 304 301 2011-08-16T13:20:48Z JoergHoletschek 3 /* Running the Setup Script */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] All of the questions do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Import configuration from existing installation:</strong> Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.foobar.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.foobar.org/biocase'', for example. After answering these three questions, the script will update your files. At the end of the output it will print the lines you will need later to create the link, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so will compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.foobar.org/biocase''. If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can a pain; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the DBMS used by the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to he library test page that shows the status of installed libraries. On a vanilla BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries ship with the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' in the browser window), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown [[Installation#Linking_BioCASe_and_Your_Web_Server|here]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] a8160cedbf1e81c1198653423c1ffe554c2ad2f9 305 304 2011-08-16T13:22:47Z JoergHoletschek 3 /* Running the Setup Script */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] All of the questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Import configuration from existing installation:</strong> Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.foobar.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.foobar.org/biocase'', for example. After answering these three questions, the script will update your files. At the end of the output it will print the lines you will need later to create the link, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so will compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.foobar.org/biocase''. If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can a pain; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the DBMS used by the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to he library test page that shows the status of installed libraries. On a vanilla BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries ship with the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' in the browser window), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown [[Installation#Linking_BioCASe_and_Your_Web_Server|here]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] 32b4d0f12ee551ae434611e803d78d89fb3d447e 306 305 2011-08-16T13:25:34Z JoergHoletschek 3 /* Running the Setup Script */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] All of the questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Import configuration from existing installation:</strong> Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.foobar.org/biocase'', for example. After answering these three questions, the script will update your files. At the end of the output it will print the lines you will need later to create the link, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so will compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.foobar.org/biocase''. If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can a pain; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the DBMS used by the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to he library test page that shows the status of installed libraries. On a vanilla BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries ship with the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' in the browser window), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown [[Installation#Linking_BioCASe_and_Your_Web_Server|here]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] dbb1ab22dbeeae94e1c3822ccedd307bec402605 307 306 2011-08-16T13:27:07Z JoergHoletschek 3 wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] All of the questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Import configuration from existing installation:</strong> Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these three questions, the script will update your files. At the end of the output it will print the lines you will need later to create the link, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so will compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can a pain; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the DBMS used by the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to he library test page that shows the status of installed libraries. On a vanilla BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries ship with the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' in the browser window), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown [[Installation#Linking_BioCASe_and_Your_Web_Server|here]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] cbfe0894f2a6e034dd6459432c3a932bb7685dcd 308 307 2011-08-16T13:30:58Z JoergHoletschek 3 /* Running the Setup Script */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] All of the questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Import configuration from existing installation:</strong> Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these three questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so will compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can a pain; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the DBMS used by the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to he library test page that shows the status of installed libraries. On a vanilla BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries ship with the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' in the browser window), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown [[Installation#Linking_BioCASe_and_Your_Web_Server|here]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] 07c9e0f522a4c08becb4fd063135d0e5622f3f2d 309 308 2011-08-16T13:31:43Z JoergHoletschek 3 /* Linking BioCASe and Your Web Server */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] All of the questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Import configuration from existing installation:</strong> Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these three questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can a pain; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the DBMS used by the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to he library test page that shows the status of installed libraries. On a vanilla BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries ship with the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' in the browser window), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown [[Installation#Linking_BioCASe_and_Your_Web_Server|here]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] db1918b64bf4fc4752b9f195c3304b2970034391 310 309 2011-08-16T13:32:55Z JoergHoletschek 3 /* Linking BioCASe and Your Web Server */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] All of the questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Import configuration from existing installation:</strong> Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these three questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can a pain; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the DBMS used by the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to he library test page that shows the status of installed libraries. On a vanilla BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries ship with the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' in the browser window), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown [[Installation#Linking_BioCASe_and_Your_Web_Server|here]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] 47ba4ead14f81523299570ec0c56ad38915c1fe8 311 310 2011-08-16T13:34:11Z JoergHoletschek 3 /* Installing Additional Python Packages */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] All of the questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Import configuration from existing installation:</strong> Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these three questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can a pain; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the DBMS used by the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a vanilla BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries ship with the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' in the browser window), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown [[Installation#Linking_BioCASe_and_Your_Web_Server|here]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] 615b6702fe655e2d13826e43d78297a264597671 0 6 312 311 2011-08-16T13:36:14Z JoergHoletschek 3 /* Installing Additional Python Packages */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] All of the questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Import configuration from existing installation:</strong> Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these three questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can a pain; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries ship with the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' in the browser window), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown [[Installation#Linking_BioCASe_and_Your_Web_Server|here]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] e235d74f8b023e86b2f722e8fe97d41ca5b198c5 313 312 2011-08-16T13:38:02Z JoergHoletschek 3 /* Installing Additional Python Packages */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] All of the questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Import configuration from existing installation:</strong> Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these three questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can a pain; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' in the browser window), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown [[Installation#Linking_BioCASe_and_Your_Web_Server|here]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] 5a7221efd72145947beb843825cfb2e0282bd252 314 313 2011-08-16T13:41:33Z JoergHoletschek 3 /* Changing the Default Password */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] All of the questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Import configuration from existing installation:</strong> Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these three questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can a pain; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown [[Installation#Linking_BioCASe_and_Your_Web_Server|here]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] e4c7e61e1d0f7af682ea32a7abf1f22fac795e10 315 314 2011-08-16T13:45:17Z JoergHoletschek 3 /* Checking Your Installation */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] All of the questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Import configuration from existing installation:</strong> Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these three questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can a pain; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] c84b59bb579ce91af8758002e93e84b5fd91d960 316 315 2011-08-16T13:45:40Z JoergHoletschek 3 /* Checking Your Installation */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] All of the questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Import configuration from existing installation:</strong> Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these three questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can a pain; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] bd8821593dafa5233774c6d8c7ba7971ab2f8a52 317 316 2011-08-16T13:46:22Z JoergHoletschek 3 /* Checking Your Installation */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] All of the questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Import configuration from existing installation:</strong> Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these three questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can a pain; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] c84b59bb579ce91af8758002e93e84b5fd91d960 318 317 2011-08-16T13:47:12Z JoergHoletschek 3 /* Checking Your Installation */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] All of the questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Import configuration from existing installation:</strong> Not implemented yet. Sorry. But this is easy to overcome: Just keep the defaults, then copy the configuration files manually. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these three questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can a pain; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a Postgres and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] 23cad608aa298afeef5802a92e9e1f395276bf95 343 318 2011-08-17T10:26:20Z JoergHoletschek 3 /* Running the Setup Script */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can a pain; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the configuration How To about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a Postgres and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] 502424c88847bda5045b056bea912b7db878480d 0 24 319 169 2011-08-16T13:48:39Z JoergHoletschek 3 /* Creating a New Data Source */ wikitext text/x-wiki This tutorial explains in detail how to set up a datasource. == A BioCASe Data Source == Publishing a database with BioCASe means it will be made accessible through a BioCASe web service, which will be used later by the biodiversity network(s) you want to join to access to your data. For example, the GBIF Harvester will send a BioCASe Scan request to that web service to get a list of all scientific names kept in the database, and subsequently several BioCASe Search requests to load all records for these names. For each database you want to publish, you need to set up a BioCASe data source (do not mix that up with ODBC data sources on Windows machines). The resulting BioCASe web service is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source. So if you made your installation available at http://www.foobar.org/biocase during the installation process and set up a data source named Herbar, the URL of the BioCASe web service would be http://www.foobar.org/biocase/pywrapper.cgi?dsa=Herbar (data source names can be case sensitive, depending on your server’s operating system). == Creating a New Data Source == The start page of the BPS will always list the data sources currently configured, that means the web services offered by that BioCASe installation. On a fresh installation, it will be empty: [[File:instBiocaseStartpage.png|500px]] Creating a new data source can be done in the configuration tool. Choose <small>Config Tool</small> on the start page, then <small>System Administration</small>. Log in with the password you set during the installation process (if you didn’t change that yet, the default is ''ACDC''). In the configuration screen, the sections <small>Data Sources</small> and <small>Recycle Bin</small> can be used to create, remove und un-remove a data source: [[File:dsaCreateDatasourceEmpty.png|500px]] For now, we want to create a new, empty data source. Enter a name – without any spaces – in the text box and click <small>Create DSA</small> (remember that the datasource name is case sensitive on Linux machines). After a second, it will appear in the list of existing data sources with a clickable link. By doing this, you created the first BioCASe web service for your installation. Clicking <small>Remove</small> will uninstall this service again (putting it in the recycle bin, so you can restore it later). Tip: If you select <small>abcdmetadata</small> in the template drop-down list, your data source will be created with an ABCD mapping for all the metadata elements stored in the metadata table mentioned in the [[Preparation]] guide. So if you did use the templates in ''sampledb/abcdmetadata'' to create the metadata table in your database, you can use this template to spare the hassle of mapping this table and start right away with your occurrence data. == Setting up the database connection == Clicking on the data source you’ve just created will take you to the datasource configuration tool. Note that the <small>Schemas</small> section will be empty if you’ve created an empty data source or list two schemas (ABCD 1.2 and 2.06) if you created the datasource based on the <small>abcdmetadata</small> template: [[File:dsaOverview.png|450px]] The link in the <small>Database Connection</small> section will take you to the connection parameters dialog. Fill in the values for your database, press <small>Save</small> and hope for the connection to turn into a nice green <small>OK</small>. '''DBMS:''' Select the entry appropriate for your collection. The BPS can directly connect to database servers (MySQL, Postgres, DB2, SQL Server, and Oracle) and to Desktop-DBMS like Access and FoxPro through ODBC. ODBC is only available on Windows, so if you plan to use it, BioCASe must be installed on a Windows Server. '''Host:''' Enter the IP address of the database server. If you’re connection through ODBC, you must leave this empty. Instead, use Windows’ ODBC Administrator to set up a '''System datasource''' (!); the server or file name of the database will be stored in the ODBC configuration. '''Database:''' Enter the name of the ODBC datasource if you’re using ODBC, the name of the database if you’re directly connecting to a server. '''User/Password:''' Database credentials to be used. As written in the [[Preparation]] guide, you should create a user with just read access for this. '''Encoding:''' The encoding of your database. If you’re not sure, leave the default, you can change it if you run into problems later. If you know your database is using a different encoding than latin-1 (which is typically the default for western languages), maybe UTF-8, select the appropriate entry in the drop-down list. [[File:dsaDBConnection.png|450px]] Once you’ve entered the correct values, press <small>Save</small>. Click <small>Overview</small> in the link list at the top of the page to return to the datasource configuration page. If the connection status doesn’t turn into <small>OK</small> (and doesn’t after you’ve returned to the overview page), please read the [[Debugging]] guide about how to trace the problem. == Setting up the Database Structure == Clicking on <small>Edit DB Structure</small> on the datasource overview page will show the DB structure setup. Make sure the connection status at the top of the page still is <small>OK</small>. In a first step, you will add all tables/views that hold information you want to publish with BioCASe. You can ignore all database tables storing only data you don’t plan to map to ABCD. Same holds true if you use cache tables (as explained in the [[Preparation]] guide): Only the cache tables need to be set up on the database structure page. In a second step, you will specify the foreign key references between these tables. === Adding Tables/Views with their Primary Keys === To add a table or view, just select it in the drop-down list. The name of the table will be filled in the <small>Alias</small> text box (you will use this alias later in the mapping process), feel free to change it into something that describes the contents of that tables (for example ''specimen'' for the root specimen table). Press <small>Add</small> to insert the table into the list. Before continuing to the next table, select the primary key in the list of attributes under <small>Primary Key Attribute(s)</small>. In case the data type is a string and not an integer, select <small>text</small> from the drop-down list. Press <small>Add</small> to add the primary key. If the primary key consists of several columns, you can add several primary key attributes. Redo this step (adding the tables first, then the primary key(s)) for all tables you want to publish data from. Always remember to set the correct primary key column(s), even if they’re not defined as primary keys in the database. Tables without a primary key cannot be used by the Provider Software and will be removed upon saving! === Specifying Foreign Key References === Once you’ve added all tables, you can define the relations between these tables by defining foreign keys. In order to do this, go to the <small>Foreign Key(s)</small> column of the table that holds a foreign key reference to another table, select the alias of the referenced table and press <small>Add</small>. Select the attribute that stores the foreign key and change the data type if it is not integer, then press <small>Add</small>. You must specify as many attributes for the foreign keys as the primary key of the referenced table has, otherwise the Provider Software won’t be able to create a valid join between the tables. Also, they must be in the same order and have identical data types. Redo this for all tables until you’ve specified all foreign key references. If you’ve installed the Graphviz package, the Provider Software will create a graph visualizing the current database structure. Each table will be visible as a node with the primary key(s) listed. References between tables will be represented by arrows, the foreign key(s) listed in the referring table: [[File:dsaDatabaseStructure_AlgenEngels.png|800px]] === Saving, Editing an Existing Setup === You can change aliases of tables you’ve already added to the list. To do so, go to the <small>Alias</small> text box of that table and type in a different name. Upon exiting the text box, all references to this alias will be updated in the table setup; also the graph will reflect the changes. But caution: Schema mappings use these aliases to bind schema elements to tables and columns, if you change an alias you’ve already used in a schema mapping, you will break this binding! So it is advisable to decide for meaningful alias names and keep them stable once you’ve started the mapping process. Once you’ve added a table/view to the table list, you can’t change the underlying table, since this would invalidate all the primary and foreign key attributes. If you’ve added the wrong table, remove it by using the trash can symbol next to <small>Delete Alias</small> and add the correct table/view. If you’ve finished the table setup and the graph displays the model of your database correctly, save the configuration by pressing <small>Save</small> at the bottom of the page. This is important; navigating away from the page without saving will cause your changes to be lost (even though you will be asked for confirmation)! It is advisable to save the table setup every now and then, for example once you’ve added all tables and primary keys. If you edit an existing table setup and happen to realize you’ve messed it up, you can revert to the state when you’ve pressed Save for the last time by using the <small>Revert</small> button at the bottom. But be careful: All unsaved changes will be lost. '''<u>Important Note:</u>''' The retrieval of tables/view/column names is currently only supported for the following DBMS: Microsoft SQL Server, MySQL, Access, Postgres and FoxPro. If you use a different DBMS, you will see a simple text box instead of a drop-down-list, where you’ll have to type in the name of the table or column you want to add. 762c5aaf8011058de1a2540882628b0d1a6f26c2 320 319 2011-08-16T13:49:26Z JoergHoletschek 3 /* Creating a New Data Source */ wikitext text/x-wiki This tutorial explains in detail how to set up a datasource. == A BioCASe Data Source == Publishing a database with BioCASe means it will be made accessible through a BioCASe web service, which will be used later by the biodiversity network(s) you want to join to access to your data. For example, the GBIF Harvester will send a BioCASe Scan request to that web service to get a list of all scientific names kept in the database, and subsequently several BioCASe Search requests to load all records for these names. For each database you want to publish, you need to set up a BioCASe data source (do not mix that up with ODBC data sources on Windows machines). The resulting BioCASe web service is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source. So if you made your installation available at http://www.foobar.org/biocase during the installation process and set up a data source named Herbar, the URL of the BioCASe web service would be http://www.foobar.org/biocase/pywrapper.cgi?dsa=Herbar (data source names can be case sensitive, depending on your server’s operating system). == Creating a New Data Source == The start page of the BPS will always list the data sources currently configured, that means the web services offered by that BioCASe installation. On a fresh installation, it will be empty: [[File:instBiocaseStartpage.png|500px]] Creating a new data source can be done in the configuration tool. Choose <small>Config Tool</small> on the start page, then <small>System Administration</small>. Log in with the password you set during the installation process (if you didn’t change that yet, the default is ''ACDC''). In the configuration screen, the sections <small>Data Sources</small> and <small>Recycle Bin</small> can be used to create, remove und restore a data source: [[File:dsaCreateDatasourceEmpty.png|500px]] For now, we want to create a new, empty data source. Enter a name – without any spaces – in the text box and click <small>Create DSA</small> (remember that the datasource name is case sensitive on Linux machines). After a second, it will appear in the list of existing data sources with a clickable link. By doing this, you created the first BioCASe web service for your installation. Clicking <small>Remove</small> will uninstall this service again (putting it in the recycle bin, so you can restore it later). Tip: If you select <small>abcdmetadata</small> in the template drop-down list, your data source will be created with an ABCD mapping for all the metadata elements stored in the metadata table mentioned in the [[Preparation]] guide. So if you did use the templates in ''sampledb/abcdmetadata'' to create the metadata table in your database, you can use this template to spare the hassle of mapping this table and start right away with your occurrence data. == Setting up the database connection == Clicking on the data source you’ve just created will take you to the datasource configuration tool. Note that the <small>Schemas</small> section will be empty if you’ve created an empty data source or list two schemas (ABCD 1.2 and 2.06) if you created the datasource based on the <small>abcdmetadata</small> template: [[File:dsaOverview.png|450px]] The link in the <small>Database Connection</small> section will take you to the connection parameters dialog. Fill in the values for your database, press <small>Save</small> and hope for the connection to turn into a nice green <small>OK</small>. '''DBMS:''' Select the entry appropriate for your collection. The BPS can directly connect to database servers (MySQL, Postgres, DB2, SQL Server, and Oracle) and to Desktop-DBMS like Access and FoxPro through ODBC. ODBC is only available on Windows, so if you plan to use it, BioCASe must be installed on a Windows Server. '''Host:''' Enter the IP address of the database server. If you’re connection through ODBC, you must leave this empty. Instead, use Windows’ ODBC Administrator to set up a '''System datasource''' (!); the server or file name of the database will be stored in the ODBC configuration. '''Database:''' Enter the name of the ODBC datasource if you’re using ODBC, the name of the database if you’re directly connecting to a server. '''User/Password:''' Database credentials to be used. As written in the [[Preparation]] guide, you should create a user with just read access for this. '''Encoding:''' The encoding of your database. If you’re not sure, leave the default, you can change it if you run into problems later. If you know your database is using a different encoding than latin-1 (which is typically the default for western languages), maybe UTF-8, select the appropriate entry in the drop-down list. [[File:dsaDBConnection.png|450px]] Once you’ve entered the correct values, press <small>Save</small>. Click <small>Overview</small> in the link list at the top of the page to return to the datasource configuration page. If the connection status doesn’t turn into <small>OK</small> (and doesn’t after you’ve returned to the overview page), please read the [[Debugging]] guide about how to trace the problem. == Setting up the Database Structure == Clicking on <small>Edit DB Structure</small> on the datasource overview page will show the DB structure setup. Make sure the connection status at the top of the page still is <small>OK</small>. In a first step, you will add all tables/views that hold information you want to publish with BioCASe. You can ignore all database tables storing only data you don’t plan to map to ABCD. Same holds true if you use cache tables (as explained in the [[Preparation]] guide): Only the cache tables need to be set up on the database structure page. In a second step, you will specify the foreign key references between these tables. === Adding Tables/Views with their Primary Keys === To add a table or view, just select it in the drop-down list. The name of the table will be filled in the <small>Alias</small> text box (you will use this alias later in the mapping process), feel free to change it into something that describes the contents of that tables (for example ''specimen'' for the root specimen table). Press <small>Add</small> to insert the table into the list. Before continuing to the next table, select the primary key in the list of attributes under <small>Primary Key Attribute(s)</small>. In case the data type is a string and not an integer, select <small>text</small> from the drop-down list. Press <small>Add</small> to add the primary key. If the primary key consists of several columns, you can add several primary key attributes. Redo this step (adding the tables first, then the primary key(s)) for all tables you want to publish data from. Always remember to set the correct primary key column(s), even if they’re not defined as primary keys in the database. Tables without a primary key cannot be used by the Provider Software and will be removed upon saving! === Specifying Foreign Key References === Once you’ve added all tables, you can define the relations between these tables by defining foreign keys. In order to do this, go to the <small>Foreign Key(s)</small> column of the table that holds a foreign key reference to another table, select the alias of the referenced table and press <small>Add</small>. Select the attribute that stores the foreign key and change the data type if it is not integer, then press <small>Add</small>. You must specify as many attributes for the foreign keys as the primary key of the referenced table has, otherwise the Provider Software won’t be able to create a valid join between the tables. Also, they must be in the same order and have identical data types. Redo this for all tables until you’ve specified all foreign key references. If you’ve installed the Graphviz package, the Provider Software will create a graph visualizing the current database structure. Each table will be visible as a node with the primary key(s) listed. References between tables will be represented by arrows, the foreign key(s) listed in the referring table: [[File:dsaDatabaseStructure_AlgenEngels.png|800px]] === Saving, Editing an Existing Setup === You can change aliases of tables you’ve already added to the list. To do so, go to the <small>Alias</small> text box of that table and type in a different name. Upon exiting the text box, all references to this alias will be updated in the table setup; also the graph will reflect the changes. But caution: Schema mappings use these aliases to bind schema elements to tables and columns, if you change an alias you’ve already used in a schema mapping, you will break this binding! So it is advisable to decide for meaningful alias names and keep them stable once you’ve started the mapping process. Once you’ve added a table/view to the table list, you can’t change the underlying table, since this would invalidate all the primary and foreign key attributes. If you’ve added the wrong table, remove it by using the trash can symbol next to <small>Delete Alias</small> and add the correct table/view. If you’ve finished the table setup and the graph displays the model of your database correctly, save the configuration by pressing <small>Save</small> at the bottom of the page. This is important; navigating away from the page without saving will cause your changes to be lost (even though you will be asked for confirmation)! It is advisable to save the table setup every now and then, for example once you’ve added all tables and primary keys. If you edit an existing table setup and happen to realize you’ve messed it up, you can revert to the state when you’ve pressed Save for the last time by using the <small>Revert</small> button at the bottom. But be careful: All unsaved changes will be lost. '''<u>Important Note:</u>''' The retrieval of tables/view/column names is currently only supported for the following DBMS: Microsoft SQL Server, MySQL, Access, Postgres and FoxPro. If you use a different DBMS, you will see a simple text box instead of a drop-down-list, where you’ll have to type in the name of the table or column you want to add. 536d67fa2c9e0972ed488b5b5ba04b7c1b82df18 323 320 2011-08-16T14:05:12Z JoergHoletschek 3 /* Setting up the database connection */ wikitext text/x-wiki This tutorial explains in detail how to set up a datasource. == A BioCASe Data Source == Publishing a database with BioCASe means it will be made accessible through a BioCASe web service, which will be used later by the biodiversity network(s) you want to join to access to your data. For example, the GBIF Harvester will send a BioCASe Scan request to that web service to get a list of all scientific names kept in the database, and subsequently several BioCASe Search requests to load all records for these names. For each database you want to publish, you need to set up a BioCASe data source (do not mix that up with ODBC data sources on Windows machines). The resulting BioCASe web service is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source. So if you made your installation available at http://www.foobar.org/biocase during the installation process and set up a data source named Herbar, the URL of the BioCASe web service would be http://www.foobar.org/biocase/pywrapper.cgi?dsa=Herbar (data source names can be case sensitive, depending on your server’s operating system). == Creating a New Data Source == The start page of the BPS will always list the data sources currently configured, that means the web services offered by that BioCASe installation. On a fresh installation, it will be empty: [[File:instBiocaseStartpage.png|500px]] Creating a new data source can be done in the configuration tool. Choose <small>Config Tool</small> on the start page, then <small>System Administration</small>. Log in with the password you set during the installation process (if you didn’t change that yet, the default is ''ACDC''). In the configuration screen, the sections <small>Data Sources</small> and <small>Recycle Bin</small> can be used to create, remove und restore a data source: [[File:dsaCreateDatasourceEmpty.png|500px]] For now, we want to create a new, empty data source. Enter a name – without any spaces – in the text box and click <small>Create DSA</small> (remember that the datasource name is case sensitive on Linux machines). After a second, it will appear in the list of existing data sources with a clickable link. By doing this, you created the first BioCASe web service for your installation. Clicking <small>Remove</small> will uninstall this service again (putting it in the recycle bin, so you can restore it later). Tip: If you select <small>abcdmetadata</small> in the template drop-down list, your data source will be created with an ABCD mapping for all the metadata elements stored in the metadata table mentioned in the [[Preparation]] guide. So if you did use the templates in ''sampledb/abcdmetadata'' to create the metadata table in your database, you can use this template to spare the hassle of mapping this table and start right away with your occurrence data. == Setting up the database connection == Clicking on the data source you’ve just created will take you to the datasource configuration tool. Note that the <small>Schemas</small> section will be empty if you’ve created an empty data source or list two schemas (ABCD 1.2 and 2.06) if you created the datasource based on the <small>abcdmetadata</small> template: [[File:dsaOverview.png|450px]] The link in the <small>Database Connection</small> section will take you to the connection parameters dialog. Fill in the values for your database, press <small>Save</small> and the connection status should turn into a nice green <small>OK</small>. '''DBMS:''' Select the entry appropriate for your collection. The BPS can directly connect to database servers (MySQL, Postgres, DB2, SQL Server, and Oracle) and to Desktop-DBMS like Access and FoxPro through ODBC. ODBC is only available on Windows, so if you plan to use it, BioCASe must be installed on a Windows Server. '''Host:''' Enter the IP address of the database server. If you’re connection through ODBC, you must leave this empty. Instead, use Windows’ ODBC Administrator to set up a '''System datasource''' (!); the server or file name of the database will be stored in the ODBC configuration. '''Database:''' Enter the name of the ODBC datasource if you’re using ODBC, the name of the database if you’re directly connecting to a server. '''User/Password:''' Database credentials to be used. As written in the [[Preparation]] guide, you should create a user with just read access for this. '''Encoding:''' The encoding of your database. If you’re not sure, leave the default, you can change it if you run into problems later. If you know your database is using a different encoding than latin-1 (which is typically the default for western languages), maybe UTF-8, select the appropriate entry in the drop-down list. [[File:dsaDBConnection.png|450px]] Once you’ve entered the correct values, press <small>Save</small>. Click <small>Overview</small> in the link list at the top of the page to return to the datasource configuration page. If the connection status doesn’t turn into <small>OK</small> (and doesn’t after you’ve returned to the overview page), please read the [[Debugging]] guide about how to trace the problem. == Setting up the Database Structure == Clicking on <small>Edit DB Structure</small> on the datasource overview page will show the DB structure setup. Make sure the connection status at the top of the page still is <small>OK</small>. In a first step, you will add all tables/views that hold information you want to publish with BioCASe. You can ignore all database tables storing only data you don’t plan to map to ABCD. Same holds true if you use cache tables (as explained in the [[Preparation]] guide): Only the cache tables need to be set up on the database structure page. In a second step, you will specify the foreign key references between these tables. === Adding Tables/Views with their Primary Keys === To add a table or view, just select it in the drop-down list. The name of the table will be filled in the <small>Alias</small> text box (you will use this alias later in the mapping process), feel free to change it into something that describes the contents of that tables (for example ''specimen'' for the root specimen table). Press <small>Add</small> to insert the table into the list. Before continuing to the next table, select the primary key in the list of attributes under <small>Primary Key Attribute(s)</small>. In case the data type is a string and not an integer, select <small>text</small> from the drop-down list. Press <small>Add</small> to add the primary key. If the primary key consists of several columns, you can add several primary key attributes. Redo this step (adding the tables first, then the primary key(s)) for all tables you want to publish data from. Always remember to set the correct primary key column(s), even if they’re not defined as primary keys in the database. Tables without a primary key cannot be used by the Provider Software and will be removed upon saving! === Specifying Foreign Key References === Once you’ve added all tables, you can define the relations between these tables by defining foreign keys. In order to do this, go to the <small>Foreign Key(s)</small> column of the table that holds a foreign key reference to another table, select the alias of the referenced table and press <small>Add</small>. Select the attribute that stores the foreign key and change the data type if it is not integer, then press <small>Add</small>. You must specify as many attributes for the foreign keys as the primary key of the referenced table has, otherwise the Provider Software won’t be able to create a valid join between the tables. Also, they must be in the same order and have identical data types. Redo this for all tables until you’ve specified all foreign key references. If you’ve installed the Graphviz package, the Provider Software will create a graph visualizing the current database structure. Each table will be visible as a node with the primary key(s) listed. References between tables will be represented by arrows, the foreign key(s) listed in the referring table: [[File:dsaDatabaseStructure_AlgenEngels.png|800px]] === Saving, Editing an Existing Setup === You can change aliases of tables you’ve already added to the list. To do so, go to the <small>Alias</small> text box of that table and type in a different name. Upon exiting the text box, all references to this alias will be updated in the table setup; also the graph will reflect the changes. But caution: Schema mappings use these aliases to bind schema elements to tables and columns, if you change an alias you’ve already used in a schema mapping, you will break this binding! So it is advisable to decide for meaningful alias names and keep them stable once you’ve started the mapping process. Once you’ve added a table/view to the table list, you can’t change the underlying table, since this would invalidate all the primary and foreign key attributes. If you’ve added the wrong table, remove it by using the trash can symbol next to <small>Delete Alias</small> and add the correct table/view. If you’ve finished the table setup and the graph displays the model of your database correctly, save the configuration by pressing <small>Save</small> at the bottom of the page. This is important; navigating away from the page without saving will cause your changes to be lost (even though you will be asked for confirmation)! It is advisable to save the table setup every now and then, for example once you’ve added all tables and primary keys. If you edit an existing table setup and happen to realize you’ve messed it up, you can revert to the state when you’ve pressed Save for the last time by using the <small>Revert</small> button at the bottom. But be careful: All unsaved changes will be lost. '''<u>Important Note:</u>''' The retrieval of tables/view/column names is currently only supported for the following DBMS: Microsoft SQL Server, MySQL, Access, Postgres and FoxPro. If you use a different DBMS, you will see a simple text box instead of a drop-down-list, where you’ll have to type in the name of the table or column you want to add. e986b16a8ad5ec522f0f9df83b4c0c613730ea36 324 323 2011-08-16T14:06:47Z JoergHoletschek 3 /* Setting up the database connection */ wikitext text/x-wiki This tutorial explains in detail how to set up a datasource. == A BioCASe Data Source == Publishing a database with BioCASe means it will be made accessible through a BioCASe web service, which will be used later by the biodiversity network(s) you want to join to access to your data. For example, the GBIF Harvester will send a BioCASe Scan request to that web service to get a list of all scientific names kept in the database, and subsequently several BioCASe Search requests to load all records for these names. For each database you want to publish, you need to set up a BioCASe data source (do not mix that up with ODBC data sources on Windows machines). The resulting BioCASe web service is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source. So if you made your installation available at http://www.foobar.org/biocase during the installation process and set up a data source named Herbar, the URL of the BioCASe web service would be http://www.foobar.org/biocase/pywrapper.cgi?dsa=Herbar (data source names can be case sensitive, depending on your server’s operating system). == Creating a New Data Source == The start page of the BPS will always list the data sources currently configured, that means the web services offered by that BioCASe installation. On a fresh installation, it will be empty: [[File:instBiocaseStartpage.png|500px]] Creating a new data source can be done in the configuration tool. Choose <small>Config Tool</small> on the start page, then <small>System Administration</small>. Log in with the password you set during the installation process (if you didn’t change that yet, the default is ''ACDC''). In the configuration screen, the sections <small>Data Sources</small> and <small>Recycle Bin</small> can be used to create, remove und restore a data source: [[File:dsaCreateDatasourceEmpty.png|500px]] For now, we want to create a new, empty data source. Enter a name – without any spaces – in the text box and click <small>Create DSA</small> (remember that the datasource name is case sensitive on Linux machines). After a second, it will appear in the list of existing data sources with a clickable link. By doing this, you created the first BioCASe web service for your installation. Clicking <small>Remove</small> will uninstall this service again (putting it in the recycle bin, so you can restore it later). Tip: If you select <small>abcdmetadata</small> in the template drop-down list, your data source will be created with an ABCD mapping for all the metadata elements stored in the metadata table mentioned in the [[Preparation]] guide. So if you did use the templates in ''sampledb/abcdmetadata'' to create the metadata table in your database, you can use this template to spare the hassle of mapping this table and start right away with your occurrence data. == Setting up the database connection == Clicking on the data source you’ve just created will take you to the datasource configuration tool. Note that the <small>Schemas</small> section will be empty if you’ve created an empty data source or list two schemas (ABCD 1.2 and 2.06) if you created the datasource based on the <small>abcdmetadata</small> template: [[File:dsaOverview.png|450px]] The link in the <small>Database Connection</small> section will take you to the connection parameters dialog. Fill in the values for your database, press <small>Save</small> and the connection status should turn into a nice green <small>OK</small>. '''DBMS:''' Select the entry appropriate for your collection. The BPS can directly connect to database servers (MySQL, Postgres, DB2, SQL Server, and Oracle) and to Desktop-DBMS like Access and FoxPro through ODBC. ODBC is only available on Windows, so if you plan to use it, BioCASe must be installed on a Windows Server. '''Host:''' Enter the IP address of the database server. If you’re connection through ODBC, you must leave this empty. Instead, use Windows’ ODBC Administrator to set up a System (!) datasource; the server or file name of the database will be stored in the ODBC configuration. '''Database:''' Enter the name of the ODBC datasource if you’re using ODBC, the name of the database if you’re directly connecting to a server. '''User/Password:''' Database credentials to be used. As written in the [[Preparation]] guide, you should create a user with just read access for this. '''Encoding:''' The encoding of your database. If you’re not sure, leave the default, you can change it if you run into problems later. If you know your database is using a different encoding than latin-1 (which is typically the default for western languages), maybe UTF-8, select the appropriate entry in the drop-down list. [[File:dsaDBConnection.png|450px]] Once you’ve entered the correct values, press <small>Save</small>. Click <small>Overview</small> in the link list at the top of the page to return to the datasource configuration page. If the connection status doesn’t turn into <small>OK</small> (and doesn’t after you’ve returned to the overview page), please read the [[Debugging]] guide about how to trace the problem. == Setting up the Database Structure == Clicking on <small>Edit DB Structure</small> on the datasource overview page will show the DB structure setup. Make sure the connection status at the top of the page still is <small>OK</small>. In a first step, you will add all tables/views that hold information you want to publish with BioCASe. You can ignore all database tables storing only data you don’t plan to map to ABCD. Same holds true if you use cache tables (as explained in the [[Preparation]] guide): Only the cache tables need to be set up on the database structure page. In a second step, you will specify the foreign key references between these tables. === Adding Tables/Views with their Primary Keys === To add a table or view, just select it in the drop-down list. The name of the table will be filled in the <small>Alias</small> text box (you will use this alias later in the mapping process), feel free to change it into something that describes the contents of that tables (for example ''specimen'' for the root specimen table). Press <small>Add</small> to insert the table into the list. Before continuing to the next table, select the primary key in the list of attributes under <small>Primary Key Attribute(s)</small>. In case the data type is a string and not an integer, select <small>text</small> from the drop-down list. Press <small>Add</small> to add the primary key. If the primary key consists of several columns, you can add several primary key attributes. Redo this step (adding the tables first, then the primary key(s)) for all tables you want to publish data from. Always remember to set the correct primary key column(s), even if they’re not defined as primary keys in the database. Tables without a primary key cannot be used by the Provider Software and will be removed upon saving! === Specifying Foreign Key References === Once you’ve added all tables, you can define the relations between these tables by defining foreign keys. In order to do this, go to the <small>Foreign Key(s)</small> column of the table that holds a foreign key reference to another table, select the alias of the referenced table and press <small>Add</small>. Select the attribute that stores the foreign key and change the data type if it is not integer, then press <small>Add</small>. You must specify as many attributes for the foreign keys as the primary key of the referenced table has, otherwise the Provider Software won’t be able to create a valid join between the tables. Also, they must be in the same order and have identical data types. Redo this for all tables until you’ve specified all foreign key references. If you’ve installed the Graphviz package, the Provider Software will create a graph visualizing the current database structure. Each table will be visible as a node with the primary key(s) listed. References between tables will be represented by arrows, the foreign key(s) listed in the referring table: [[File:dsaDatabaseStructure_AlgenEngels.png|800px]] === Saving, Editing an Existing Setup === You can change aliases of tables you’ve already added to the list. To do so, go to the <small>Alias</small> text box of that table and type in a different name. Upon exiting the text box, all references to this alias will be updated in the table setup; also the graph will reflect the changes. But caution: Schema mappings use these aliases to bind schema elements to tables and columns, if you change an alias you’ve already used in a schema mapping, you will break this binding! So it is advisable to decide for meaningful alias names and keep them stable once you’ve started the mapping process. Once you’ve added a table/view to the table list, you can’t change the underlying table, since this would invalidate all the primary and foreign key attributes. If you’ve added the wrong table, remove it by using the trash can symbol next to <small>Delete Alias</small> and add the correct table/view. If you’ve finished the table setup and the graph displays the model of your database correctly, save the configuration by pressing <small>Save</small> at the bottom of the page. This is important; navigating away from the page without saving will cause your changes to be lost (even though you will be asked for confirmation)! It is advisable to save the table setup every now and then, for example once you’ve added all tables and primary keys. If you edit an existing table setup and happen to realize you’ve messed it up, you can revert to the state when you’ve pressed Save for the last time by using the <small>Revert</small> button at the bottom. But be careful: All unsaved changes will be lost. '''<u>Important Note:</u>''' The retrieval of tables/view/column names is currently only supported for the following DBMS: Microsoft SQL Server, MySQL, Access, Postgres and FoxPro. If you use a different DBMS, you will see a simple text box instead of a drop-down-list, where you’ll have to type in the name of the table or column you want to add. 37306dff37ceb8bbc0e18e23fb6e9c653c3d80aa 325 324 2011-08-16T14:07:51Z JoergHoletschek 3 /* Setting up the database connection */ wikitext text/x-wiki This tutorial explains in detail how to set up a datasource. == A BioCASe Data Source == Publishing a database with BioCASe means it will be made accessible through a BioCASe web service, which will be used later by the biodiversity network(s) you want to join to access to your data. For example, the GBIF Harvester will send a BioCASe Scan request to that web service to get a list of all scientific names kept in the database, and subsequently several BioCASe Search requests to load all records for these names. For each database you want to publish, you need to set up a BioCASe data source (do not mix that up with ODBC data sources on Windows machines). The resulting BioCASe web service is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source. So if you made your installation available at http://www.foobar.org/biocase during the installation process and set up a data source named Herbar, the URL of the BioCASe web service would be http://www.foobar.org/biocase/pywrapper.cgi?dsa=Herbar (data source names can be case sensitive, depending on your server’s operating system). == Creating a New Data Source == The start page of the BPS will always list the data sources currently configured, that means the web services offered by that BioCASe installation. On a fresh installation, it will be empty: [[File:instBiocaseStartpage.png|500px]] Creating a new data source can be done in the configuration tool. Choose <small>Config Tool</small> on the start page, then <small>System Administration</small>. Log in with the password you set during the installation process (if you didn’t change that yet, the default is ''ACDC''). In the configuration screen, the sections <small>Data Sources</small> and <small>Recycle Bin</small> can be used to create, remove und restore a data source: [[File:dsaCreateDatasourceEmpty.png|500px]] For now, we want to create a new, empty data source. Enter a name – without any spaces – in the text box and click <small>Create DSA</small> (remember that the datasource name is case sensitive on Linux machines). After a second, it will appear in the list of existing data sources with a clickable link. By doing this, you created the first BioCASe web service for your installation. Clicking <small>Remove</small> will uninstall this service again (putting it in the recycle bin, so you can restore it later). Tip: If you select <small>abcdmetadata</small> in the template drop-down list, your data source will be created with an ABCD mapping for all the metadata elements stored in the metadata table mentioned in the [[Preparation]] guide. So if you did use the templates in ''sampledb/abcdmetadata'' to create the metadata table in your database, you can use this template to spare the hassle of mapping this table and start right away with your occurrence data. == Setting up the database connection == Clicking on the data source you’ve just created will take you to the datasource configuration tool. Note that the <small>Schemas</small> section will be empty if you’ve created an empty data source or list two schemas (ABCD 1.2 and 2.06) if you created the datasource based on the <small>abcdmetadata</small> template: [[File:dsaOverview.png|450px]] The link in the <small>Database Connection</small> section will take you to the connection parameters dialog. Fill in the values for your database, press <small>Save</small> and the connection status should turn into a nice green <small>OK</small>. '''DBMS:''' Select the entry appropriate for your collection. The BPS can directly connect to database servers (MySQL, Postgres, DB2, SQL Server, and Oracle) and to Desktop-DBMS like Access and FoxPro through ODBC. ODBC is only available on Windows, so if you plan to use it, BioCASe must be installed on a Windows Server. '''Host:''' Enter the IP address of the database server. If you’re connecting through ODBC, you must leave this empty. Instead, use Windows’ ODBC Administrator to set up a System (!) datasource; the server or file name of the database will be stored in the ODBC configuration. '''Database:''' Enter the name of the ODBC datasource if you’re using ODBC, the name of the database if you’re directly connecting to a server. '''User/Password:''' Database credentials to be used. As written in the [[Preparation]] guide, you should create a user with just read access for this. '''Encoding:''' The encoding of your database. If you’re not sure, leave the default, you can change it if you run into problems later. If you know your database is using a different encoding than latin-1 (which is typically the default for western languages), maybe UTF-8, select the appropriate entry in the drop-down list. [[File:dsaDBConnection.png|450px]] Once you’ve entered the correct values, press <small>Save</small>. Click <small>Overview</small> in the link list at the top of the page to return to the datasource configuration page. If the connection status doesn’t turn into <small>OK</small> (and doesn’t after you’ve returned to the overview page), please read the [[Debugging]] guide about how to trace the problem. == Setting up the Database Structure == Clicking on <small>Edit DB Structure</small> on the datasource overview page will show the DB structure setup. Make sure the connection status at the top of the page still is <small>OK</small>. In a first step, you will add all tables/views that hold information you want to publish with BioCASe. You can ignore all database tables storing only data you don’t plan to map to ABCD. Same holds true if you use cache tables (as explained in the [[Preparation]] guide): Only the cache tables need to be set up on the database structure page. In a second step, you will specify the foreign key references between these tables. === Adding Tables/Views with their Primary Keys === To add a table or view, just select it in the drop-down list. The name of the table will be filled in the <small>Alias</small> text box (you will use this alias later in the mapping process), feel free to change it into something that describes the contents of that tables (for example ''specimen'' for the root specimen table). Press <small>Add</small> to insert the table into the list. Before continuing to the next table, select the primary key in the list of attributes under <small>Primary Key Attribute(s)</small>. In case the data type is a string and not an integer, select <small>text</small> from the drop-down list. Press <small>Add</small> to add the primary key. If the primary key consists of several columns, you can add several primary key attributes. Redo this step (adding the tables first, then the primary key(s)) for all tables you want to publish data from. Always remember to set the correct primary key column(s), even if they’re not defined as primary keys in the database. Tables without a primary key cannot be used by the Provider Software and will be removed upon saving! === Specifying Foreign Key References === Once you’ve added all tables, you can define the relations between these tables by defining foreign keys. In order to do this, go to the <small>Foreign Key(s)</small> column of the table that holds a foreign key reference to another table, select the alias of the referenced table and press <small>Add</small>. Select the attribute that stores the foreign key and change the data type if it is not integer, then press <small>Add</small>. You must specify as many attributes for the foreign keys as the primary key of the referenced table has, otherwise the Provider Software won’t be able to create a valid join between the tables. Also, they must be in the same order and have identical data types. Redo this for all tables until you’ve specified all foreign key references. If you’ve installed the Graphviz package, the Provider Software will create a graph visualizing the current database structure. Each table will be visible as a node with the primary key(s) listed. References between tables will be represented by arrows, the foreign key(s) listed in the referring table: [[File:dsaDatabaseStructure_AlgenEngels.png|800px]] === Saving, Editing an Existing Setup === You can change aliases of tables you’ve already added to the list. To do so, go to the <small>Alias</small> text box of that table and type in a different name. Upon exiting the text box, all references to this alias will be updated in the table setup; also the graph will reflect the changes. But caution: Schema mappings use these aliases to bind schema elements to tables and columns, if you change an alias you’ve already used in a schema mapping, you will break this binding! So it is advisable to decide for meaningful alias names and keep them stable once you’ve started the mapping process. Once you’ve added a table/view to the table list, you can’t change the underlying table, since this would invalidate all the primary and foreign key attributes. If you’ve added the wrong table, remove it by using the trash can symbol next to <small>Delete Alias</small> and add the correct table/view. If you’ve finished the table setup and the graph displays the model of your database correctly, save the configuration by pressing <small>Save</small> at the bottom of the page. This is important; navigating away from the page without saving will cause your changes to be lost (even though you will be asked for confirmation)! It is advisable to save the table setup every now and then, for example once you’ve added all tables and primary keys. If you edit an existing table setup and happen to realize you’ve messed it up, you can revert to the state when you’ve pressed Save for the last time by using the <small>Revert</small> button at the bottom. But be careful: All unsaved changes will be lost. '''<u>Important Note:</u>''' The retrieval of tables/view/column names is currently only supported for the following DBMS: Microsoft SQL Server, MySQL, Access, Postgres and FoxPro. If you use a different DBMS, you will see a simple text box instead of a drop-down-list, where you’ll have to type in the name of the table or column you want to add. 434da65dd240a6a6cb62e0800f826c50062d37ba 326 325 2011-08-16T14:08:26Z JoergHoletschek 3 /* Setting up the database connection */ wikitext text/x-wiki This tutorial explains in detail how to set up a datasource. == A BioCASe Data Source == Publishing a database with BioCASe means it will be made accessible through a BioCASe web service, which will be used later by the biodiversity network(s) you want to join to access to your data. For example, the GBIF Harvester will send a BioCASe Scan request to that web service to get a list of all scientific names kept in the database, and subsequently several BioCASe Search requests to load all records for these names. For each database you want to publish, you need to set up a BioCASe data source (do not mix that up with ODBC data sources on Windows machines). The resulting BioCASe web service is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source. So if you made your installation available at http://www.foobar.org/biocase during the installation process and set up a data source named Herbar, the URL of the BioCASe web service would be http://www.foobar.org/biocase/pywrapper.cgi?dsa=Herbar (data source names can be case sensitive, depending on your server’s operating system). == Creating a New Data Source == The start page of the BPS will always list the data sources currently configured, that means the web services offered by that BioCASe installation. On a fresh installation, it will be empty: [[File:instBiocaseStartpage.png|500px]] Creating a new data source can be done in the configuration tool. Choose <small>Config Tool</small> on the start page, then <small>System Administration</small>. Log in with the password you set during the installation process (if you didn’t change that yet, the default is ''ACDC''). In the configuration screen, the sections <small>Data Sources</small> and <small>Recycle Bin</small> can be used to create, remove und restore a data source: [[File:dsaCreateDatasourceEmpty.png|500px]] For now, we want to create a new, empty data source. Enter a name – without any spaces – in the text box and click <small>Create DSA</small> (remember that the datasource name is case sensitive on Linux machines). After a second, it will appear in the list of existing data sources with a clickable link. By doing this, you created the first BioCASe web service for your installation. Clicking <small>Remove</small> will uninstall this service again (putting it in the recycle bin, so you can restore it later). Tip: If you select <small>abcdmetadata</small> in the template drop-down list, your data source will be created with an ABCD mapping for all the metadata elements stored in the metadata table mentioned in the [[Preparation]] guide. So if you did use the templates in ''sampledb/abcdmetadata'' to create the metadata table in your database, you can use this template to spare the hassle of mapping this table and start right away with your occurrence data. == Setting up the database connection == Clicking on the data source you’ve just created will take you to the datasource configuration tool. Note that the <small>Schemas</small> section will be empty if you’ve created an empty data source or list two schemas (ABCD 1.2 and 2.06) if you created the datasource based on the <small>abcdmetadata</small> template: [[File:dsaOverview.png|450px]] The link in the <small>Database Connection</small> section will take you to the connection parameters dialog. Fill in the values for your database, press <small>Save</small> and the connection status should turn into a nice green <small>OK</small>. '''DBMS:''' Select the entry appropriate for your collection. The BPS can directly connect to database servers (MySQL, Postgres, DB2, SQL Server, and Oracle) and to Desktop-DBMS like Access and FoxPro through ODBC. ODBC is only available on Windows, so if you plan to use it, BioCASe must be installed on a Windows Server. '''Host:''' Enter the IP address of the database server. If you’re connecting through ODBC, you must leave this empty. Instead, use Windows’ ODBC Administrator to set up a System (!) datasource; the server or file name of the database will be stored in the ODBC configuration. '''Database:''' Enter the name of the ODBC datasource if you’re using ODBC, or the name of the database if you’re directly connecting to a server. '''User/Password:''' Database credentials to be used. As written in the [[Preparation]] guide, you should create a user with just read access for this. '''Encoding:''' The encoding of your database. If you’re not sure, leave the default, you can change it if you run into problems later. If you know your database is using a different encoding than latin-1 (which is typically the default for western languages), maybe UTF-8, select the appropriate entry in the drop-down list. [[File:dsaDBConnection.png|450px]] Once you’ve entered the correct values, press <small>Save</small>. Click <small>Overview</small> in the link list at the top of the page to return to the datasource configuration page. If the connection status doesn’t turn into <small>OK</small> (and doesn’t after you’ve returned to the overview page), please read the [[Debugging]] guide about how to trace the problem. == Setting up the Database Structure == Clicking on <small>Edit DB Structure</small> on the datasource overview page will show the DB structure setup. Make sure the connection status at the top of the page still is <small>OK</small>. In a first step, you will add all tables/views that hold information you want to publish with BioCASe. You can ignore all database tables storing only data you don’t plan to map to ABCD. Same holds true if you use cache tables (as explained in the [[Preparation]] guide): Only the cache tables need to be set up on the database structure page. In a second step, you will specify the foreign key references between these tables. === Adding Tables/Views with their Primary Keys === To add a table or view, just select it in the drop-down list. The name of the table will be filled in the <small>Alias</small> text box (you will use this alias later in the mapping process), feel free to change it into something that describes the contents of that tables (for example ''specimen'' for the root specimen table). Press <small>Add</small> to insert the table into the list. Before continuing to the next table, select the primary key in the list of attributes under <small>Primary Key Attribute(s)</small>. In case the data type is a string and not an integer, select <small>text</small> from the drop-down list. Press <small>Add</small> to add the primary key. If the primary key consists of several columns, you can add several primary key attributes. Redo this step (adding the tables first, then the primary key(s)) for all tables you want to publish data from. Always remember to set the correct primary key column(s), even if they’re not defined as primary keys in the database. Tables without a primary key cannot be used by the Provider Software and will be removed upon saving! === Specifying Foreign Key References === Once you’ve added all tables, you can define the relations between these tables by defining foreign keys. In order to do this, go to the <small>Foreign Key(s)</small> column of the table that holds a foreign key reference to another table, select the alias of the referenced table and press <small>Add</small>. Select the attribute that stores the foreign key and change the data type if it is not integer, then press <small>Add</small>. You must specify as many attributes for the foreign keys as the primary key of the referenced table has, otherwise the Provider Software won’t be able to create a valid join between the tables. Also, they must be in the same order and have identical data types. Redo this for all tables until you’ve specified all foreign key references. If you’ve installed the Graphviz package, the Provider Software will create a graph visualizing the current database structure. Each table will be visible as a node with the primary key(s) listed. References between tables will be represented by arrows, the foreign key(s) listed in the referring table: [[File:dsaDatabaseStructure_AlgenEngels.png|800px]] === Saving, Editing an Existing Setup === You can change aliases of tables you’ve already added to the list. To do so, go to the <small>Alias</small> text box of that table and type in a different name. Upon exiting the text box, all references to this alias will be updated in the table setup; also the graph will reflect the changes. But caution: Schema mappings use these aliases to bind schema elements to tables and columns, if you change an alias you’ve already used in a schema mapping, you will break this binding! So it is advisable to decide for meaningful alias names and keep them stable once you’ve started the mapping process. Once you’ve added a table/view to the table list, you can’t change the underlying table, since this would invalidate all the primary and foreign key attributes. If you’ve added the wrong table, remove it by using the trash can symbol next to <small>Delete Alias</small> and add the correct table/view. If you’ve finished the table setup and the graph displays the model of your database correctly, save the configuration by pressing <small>Save</small> at the bottom of the page. This is important; navigating away from the page without saving will cause your changes to be lost (even though you will be asked for confirmation)! It is advisable to save the table setup every now and then, for example once you’ve added all tables and primary keys. If you edit an existing table setup and happen to realize you’ve messed it up, you can revert to the state when you’ve pressed Save for the last time by using the <small>Revert</small> button at the bottom. But be careful: All unsaved changes will be lost. '''<u>Important Note:</u>''' The retrieval of tables/view/column names is currently only supported for the following DBMS: Microsoft SQL Server, MySQL, Access, Postgres and FoxPro. If you use a different DBMS, you will see a simple text box instead of a drop-down-list, where you’ll have to type in the name of the table or column you want to add. b0f464ed7d145aae61fc3df956984d2a46ed8da3 327 326 2011-08-16T14:12:22Z JoergHoletschek 3 /* Adding Tables/Views with their Primary Keys */ wikitext text/x-wiki This tutorial explains in detail how to set up a datasource. == A BioCASe Data Source == Publishing a database with BioCASe means it will be made accessible through a BioCASe web service, which will be used later by the biodiversity network(s) you want to join to access to your data. For example, the GBIF Harvester will send a BioCASe Scan request to that web service to get a list of all scientific names kept in the database, and subsequently several BioCASe Search requests to load all records for these names. For each database you want to publish, you need to set up a BioCASe data source (do not mix that up with ODBC data sources on Windows machines). The resulting BioCASe web service is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source. So if you made your installation available at http://www.foobar.org/biocase during the installation process and set up a data source named Herbar, the URL of the BioCASe web service would be http://www.foobar.org/biocase/pywrapper.cgi?dsa=Herbar (data source names can be case sensitive, depending on your server’s operating system). == Creating a New Data Source == The start page of the BPS will always list the data sources currently configured, that means the web services offered by that BioCASe installation. On a fresh installation, it will be empty: [[File:instBiocaseStartpage.png|500px]] Creating a new data source can be done in the configuration tool. Choose <small>Config Tool</small> on the start page, then <small>System Administration</small>. Log in with the password you set during the installation process (if you didn’t change that yet, the default is ''ACDC''). In the configuration screen, the sections <small>Data Sources</small> and <small>Recycle Bin</small> can be used to create, remove und restore a data source: [[File:dsaCreateDatasourceEmpty.png|500px]] For now, we want to create a new, empty data source. Enter a name – without any spaces – in the text box and click <small>Create DSA</small> (remember that the datasource name is case sensitive on Linux machines). After a second, it will appear in the list of existing data sources with a clickable link. By doing this, you created the first BioCASe web service for your installation. Clicking <small>Remove</small> will uninstall this service again (putting it in the recycle bin, so you can restore it later). Tip: If you select <small>abcdmetadata</small> in the template drop-down list, your data source will be created with an ABCD mapping for all the metadata elements stored in the metadata table mentioned in the [[Preparation]] guide. So if you did use the templates in ''sampledb/abcdmetadata'' to create the metadata table in your database, you can use this template to spare the hassle of mapping this table and start right away with your occurrence data. == Setting up the database connection == Clicking on the data source you’ve just created will take you to the datasource configuration tool. Note that the <small>Schemas</small> section will be empty if you’ve created an empty data source or list two schemas (ABCD 1.2 and 2.06) if you created the datasource based on the <small>abcdmetadata</small> template: [[File:dsaOverview.png|450px]] The link in the <small>Database Connection</small> section will take you to the connection parameters dialog. Fill in the values for your database, press <small>Save</small> and the connection status should turn into a nice green <small>OK</small>. '''DBMS:''' Select the entry appropriate for your collection. The BPS can directly connect to database servers (MySQL, Postgres, DB2, SQL Server, and Oracle) and to Desktop-DBMS like Access and FoxPro through ODBC. ODBC is only available on Windows, so if you plan to use it, BioCASe must be installed on a Windows Server. '''Host:''' Enter the IP address of the database server. If you’re connecting through ODBC, you must leave this empty. Instead, use Windows’ ODBC Administrator to set up a System (!) datasource; the server or file name of the database will be stored in the ODBC configuration. '''Database:''' Enter the name of the ODBC datasource if you’re using ODBC, or the name of the database if you’re directly connecting to a server. '''User/Password:''' Database credentials to be used. As written in the [[Preparation]] guide, you should create a user with just read access for this. '''Encoding:''' The encoding of your database. If you’re not sure, leave the default, you can change it if you run into problems later. If you know your database is using a different encoding than latin-1 (which is typically the default for western languages), maybe UTF-8, select the appropriate entry in the drop-down list. [[File:dsaDBConnection.png|450px]] Once you’ve entered the correct values, press <small>Save</small>. Click <small>Overview</small> in the link list at the top of the page to return to the datasource configuration page. If the connection status doesn’t turn into <small>OK</small> (and doesn’t after you’ve returned to the overview page), please read the [[Debugging]] guide about how to trace the problem. == Setting up the Database Structure == Clicking on <small>Edit DB Structure</small> on the datasource overview page will show the DB structure setup. Make sure the connection status at the top of the page still is <small>OK</small>. In a first step, you will add all tables/views that hold information you want to publish with BioCASe. You can ignore all database tables storing only data you don’t plan to map to ABCD. Same holds true if you use cache tables (as explained in the [[Preparation]] guide): Only the cache tables need to be set up on the database structure page. In a second step, you will specify the foreign key references between these tables. === Adding Tables/Views with their Primary Keys === To add a table or view, just select it in the drop-down list. The name of the table will be filled in the <small>Alias</small> text box (you will use this alias later in the mapping process), feel free to change it into something that describes the contents of that tables (for example ''specimen'' for the root specimen table). Press <small>Add</small> to insert the table into the list. Before continuing to the next table, select the primary key in the list of attributes under <small>Primary Key Attribute(s)</small>. In case the data type is a string and not an integer, select <small>text</small> from the drop-down list. Press <small>Add</small> to add the primary key. If the primary key consists of several columns, you can add several primary key attributes. Redo this step (adding the tables first, then the primary key/keys) for all tables you want to publish data from. Always remember to set the correct primary key column(s), even if they’re not defined as primary keys in the database. Tables without a primary key cannot be used by the Provider Software and will be removed upon saving! === Specifying Foreign Key References === Once you’ve added all tables, you can define the relations between these tables by defining foreign keys. In order to do this, go to the <small>Foreign Key(s)</small> column of the table that holds a foreign key reference to another table, select the alias of the referenced table and press <small>Add</small>. Select the attribute that stores the foreign key and change the data type if it is not integer, then press <small>Add</small>. You must specify as many attributes for the foreign keys as the primary key of the referenced table has, otherwise the Provider Software won’t be able to create a valid join between the tables. Also, they must be in the same order and have identical data types. Redo this for all tables until you’ve specified all foreign key references. If you’ve installed the Graphviz package, the Provider Software will create a graph visualizing the current database structure. Each table will be visible as a node with the primary key(s) listed. References between tables will be represented by arrows, the foreign key(s) listed in the referring table: [[File:dsaDatabaseStructure_AlgenEngels.png|800px]] === Saving, Editing an Existing Setup === You can change aliases of tables you’ve already added to the list. To do so, go to the <small>Alias</small> text box of that table and type in a different name. Upon exiting the text box, all references to this alias will be updated in the table setup; also the graph will reflect the changes. But caution: Schema mappings use these aliases to bind schema elements to tables and columns, if you change an alias you’ve already used in a schema mapping, you will break this binding! So it is advisable to decide for meaningful alias names and keep them stable once you’ve started the mapping process. Once you’ve added a table/view to the table list, you can’t change the underlying table, since this would invalidate all the primary and foreign key attributes. If you’ve added the wrong table, remove it by using the trash can symbol next to <small>Delete Alias</small> and add the correct table/view. If you’ve finished the table setup and the graph displays the model of your database correctly, save the configuration by pressing <small>Save</small> at the bottom of the page. This is important; navigating away from the page without saving will cause your changes to be lost (even though you will be asked for confirmation)! It is advisable to save the table setup every now and then, for example once you’ve added all tables and primary keys. If you edit an existing table setup and happen to realize you’ve messed it up, you can revert to the state when you’ve pressed Save for the last time by using the <small>Revert</small> button at the bottom. But be careful: All unsaved changes will be lost. '''<u>Important Note:</u>''' The retrieval of tables/view/column names is currently only supported for the following DBMS: Microsoft SQL Server, MySQL, Access, Postgres and FoxPro. If you use a different DBMS, you will see a simple text box instead of a drop-down-list, where you’ll have to type in the name of the table or column you want to add. 0aa175131c2c9a6d0cf2041a75238dbed8fc0ea0 328 327 2011-08-16T14:12:53Z JoergHoletschek 3 /* Adding Tables/Views with their Primary Keys */ wikitext text/x-wiki This tutorial explains in detail how to set up a datasource. == A BioCASe Data Source == Publishing a database with BioCASe means it will be made accessible through a BioCASe web service, which will be used later by the biodiversity network(s) you want to join to access to your data. For example, the GBIF Harvester will send a BioCASe Scan request to that web service to get a list of all scientific names kept in the database, and subsequently several BioCASe Search requests to load all records for these names. For each database you want to publish, you need to set up a BioCASe data source (do not mix that up with ODBC data sources on Windows machines). The resulting BioCASe web service is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source. So if you made your installation available at http://www.foobar.org/biocase during the installation process and set up a data source named Herbar, the URL of the BioCASe web service would be http://www.foobar.org/biocase/pywrapper.cgi?dsa=Herbar (data source names can be case sensitive, depending on your server’s operating system). == Creating a New Data Source == The start page of the BPS will always list the data sources currently configured, that means the web services offered by that BioCASe installation. On a fresh installation, it will be empty: [[File:instBiocaseStartpage.png|500px]] Creating a new data source can be done in the configuration tool. Choose <small>Config Tool</small> on the start page, then <small>System Administration</small>. Log in with the password you set during the installation process (if you didn’t change that yet, the default is ''ACDC''). In the configuration screen, the sections <small>Data Sources</small> and <small>Recycle Bin</small> can be used to create, remove und restore a data source: [[File:dsaCreateDatasourceEmpty.png|500px]] For now, we want to create a new, empty data source. Enter a name – without any spaces – in the text box and click <small>Create DSA</small> (remember that the datasource name is case sensitive on Linux machines). After a second, it will appear in the list of existing data sources with a clickable link. By doing this, you created the first BioCASe web service for your installation. Clicking <small>Remove</small> will uninstall this service again (putting it in the recycle bin, so you can restore it later). Tip: If you select <small>abcdmetadata</small> in the template drop-down list, your data source will be created with an ABCD mapping for all the metadata elements stored in the metadata table mentioned in the [[Preparation]] guide. So if you did use the templates in ''sampledb/abcdmetadata'' to create the metadata table in your database, you can use this template to spare the hassle of mapping this table and start right away with your occurrence data. == Setting up the database connection == Clicking on the data source you’ve just created will take you to the datasource configuration tool. Note that the <small>Schemas</small> section will be empty if you’ve created an empty data source or list two schemas (ABCD 1.2 and 2.06) if you created the datasource based on the <small>abcdmetadata</small> template: [[File:dsaOverview.png|450px]] The link in the <small>Database Connection</small> section will take you to the connection parameters dialog. Fill in the values for your database, press <small>Save</small> and the connection status should turn into a nice green <small>OK</small>. '''DBMS:''' Select the entry appropriate for your collection. The BPS can directly connect to database servers (MySQL, Postgres, DB2, SQL Server, and Oracle) and to Desktop-DBMS like Access and FoxPro through ODBC. ODBC is only available on Windows, so if you plan to use it, BioCASe must be installed on a Windows Server. '''Host:''' Enter the IP address of the database server. If you’re connecting through ODBC, you must leave this empty. Instead, use Windows’ ODBC Administrator to set up a System (!) datasource; the server or file name of the database will be stored in the ODBC configuration. '''Database:''' Enter the name of the ODBC datasource if you’re using ODBC, or the name of the database if you’re directly connecting to a server. '''User/Password:''' Database credentials to be used. As written in the [[Preparation]] guide, you should create a user with just read access for this. '''Encoding:''' The encoding of your database. If you’re not sure, leave the default, you can change it if you run into problems later. If you know your database is using a different encoding than latin-1 (which is typically the default for western languages), maybe UTF-8, select the appropriate entry in the drop-down list. [[File:dsaDBConnection.png|450px]] Once you’ve entered the correct values, press <small>Save</small>. Click <small>Overview</small> in the link list at the top of the page to return to the datasource configuration page. If the connection status doesn’t turn into <small>OK</small> (and doesn’t after you’ve returned to the overview page), please read the [[Debugging]] guide about how to trace the problem. == Setting up the Database Structure == Clicking on <small>Edit DB Structure</small> on the datasource overview page will show the DB structure setup. Make sure the connection status at the top of the page still is <small>OK</small>. In a first step, you will add all tables/views that hold information you want to publish with BioCASe. You can ignore all database tables storing only data you don’t plan to map to ABCD. Same holds true if you use cache tables (as explained in the [[Preparation]] guide): Only the cache tables need to be set up on the database structure page. In a second step, you will specify the foreign key references between these tables. === Adding Tables/Views with their Primary Keys === To add a table or view, just select it in the drop-down list. The name of the table will be filled in the <small>Alias</small> text box (you will use this alias later in the mapping process), feel free to change it into something that describes the contents of that tables (for example ''specimen'' for the root specimen table). Press <small>Add</small> to insert the table into the list. Before continuing to the next table, select the primary key in the list of attributes under <small>Primary Key Attribute(s)</small>. In case the data type is a string and not an integer, select <small>text</small> from the drop-down list. Press <small>Add</small> to add the primary key. If the primary key consists of several columns, you can add several primary key attributes. Redo these steps (adding the tables first, then the primary key/keys) for all tables you want to publish data from. Always remember to set the correct primary key column(s), even if they’re not defined as primary keys in the database. Tables without a primary key cannot be used by the Provider Software and will be removed upon saving! === Specifying Foreign Key References === Once you’ve added all tables, you can define the relations between these tables by defining foreign keys. In order to do this, go to the <small>Foreign Key(s)</small> column of the table that holds a foreign key reference to another table, select the alias of the referenced table and press <small>Add</small>. Select the attribute that stores the foreign key and change the data type if it is not integer, then press <small>Add</small>. You must specify as many attributes for the foreign keys as the primary key of the referenced table has, otherwise the Provider Software won’t be able to create a valid join between the tables. Also, they must be in the same order and have identical data types. Redo this for all tables until you’ve specified all foreign key references. If you’ve installed the Graphviz package, the Provider Software will create a graph visualizing the current database structure. Each table will be visible as a node with the primary key(s) listed. References between tables will be represented by arrows, the foreign key(s) listed in the referring table: [[File:dsaDatabaseStructure_AlgenEngels.png|800px]] === Saving, Editing an Existing Setup === You can change aliases of tables you’ve already added to the list. To do so, go to the <small>Alias</small> text box of that table and type in a different name. Upon exiting the text box, all references to this alias will be updated in the table setup; also the graph will reflect the changes. But caution: Schema mappings use these aliases to bind schema elements to tables and columns, if you change an alias you’ve already used in a schema mapping, you will break this binding! So it is advisable to decide for meaningful alias names and keep them stable once you’ve started the mapping process. Once you’ve added a table/view to the table list, you can’t change the underlying table, since this would invalidate all the primary and foreign key attributes. If you’ve added the wrong table, remove it by using the trash can symbol next to <small>Delete Alias</small> and add the correct table/view. If you’ve finished the table setup and the graph displays the model of your database correctly, save the configuration by pressing <small>Save</small> at the bottom of the page. This is important; navigating away from the page without saving will cause your changes to be lost (even though you will be asked for confirmation)! It is advisable to save the table setup every now and then, for example once you’ve added all tables and primary keys. If you edit an existing table setup and happen to realize you’ve messed it up, you can revert to the state when you’ve pressed Save for the last time by using the <small>Revert</small> button at the bottom. But be careful: All unsaved changes will be lost. '''<u>Important Note:</u>''' The retrieval of tables/view/column names is currently only supported for the following DBMS: Microsoft SQL Server, MySQL, Access, Postgres and FoxPro. If you use a different DBMS, you will see a simple text box instead of a drop-down-list, where you’ll have to type in the name of the table or column you want to add. 6abe16b2ab5063dd48c5cfa7b700ed3475871fa9 329 328 2011-08-16T14:14:14Z JoergHoletschek 3 /* Adding Tables/Views with their Primary Keys */ wikitext text/x-wiki This tutorial explains in detail how to set up a datasource. == A BioCASe Data Source == Publishing a database with BioCASe means it will be made accessible through a BioCASe web service, which will be used later by the biodiversity network(s) you want to join to access to your data. For example, the GBIF Harvester will send a BioCASe Scan request to that web service to get a list of all scientific names kept in the database, and subsequently several BioCASe Search requests to load all records for these names. For each database you want to publish, you need to set up a BioCASe data source (do not mix that up with ODBC data sources on Windows machines). The resulting BioCASe web service is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source. So if you made your installation available at http://www.foobar.org/biocase during the installation process and set up a data source named Herbar, the URL of the BioCASe web service would be http://www.foobar.org/biocase/pywrapper.cgi?dsa=Herbar (data source names can be case sensitive, depending on your server’s operating system). == Creating a New Data Source == The start page of the BPS will always list the data sources currently configured, that means the web services offered by that BioCASe installation. On a fresh installation, it will be empty: [[File:instBiocaseStartpage.png|500px]] Creating a new data source can be done in the configuration tool. Choose <small>Config Tool</small> on the start page, then <small>System Administration</small>. Log in with the password you set during the installation process (if you didn’t change that yet, the default is ''ACDC''). In the configuration screen, the sections <small>Data Sources</small> and <small>Recycle Bin</small> can be used to create, remove und restore a data source: [[File:dsaCreateDatasourceEmpty.png|500px]] For now, we want to create a new, empty data source. Enter a name – without any spaces – in the text box and click <small>Create DSA</small> (remember that the datasource name is case sensitive on Linux machines). After a second, it will appear in the list of existing data sources with a clickable link. By doing this, you created the first BioCASe web service for your installation. Clicking <small>Remove</small> will uninstall this service again (putting it in the recycle bin, so you can restore it later). Tip: If you select <small>abcdmetadata</small> in the template drop-down list, your data source will be created with an ABCD mapping for all the metadata elements stored in the metadata table mentioned in the [[Preparation]] guide. So if you did use the templates in ''sampledb/abcdmetadata'' to create the metadata table in your database, you can use this template to spare the hassle of mapping this table and start right away with your occurrence data. == Setting up the database connection == Clicking on the data source you’ve just created will take you to the datasource configuration tool. Note that the <small>Schemas</small> section will be empty if you’ve created an empty data source or list two schemas (ABCD 1.2 and 2.06) if you created the datasource based on the <small>abcdmetadata</small> template: [[File:dsaOverview.png|450px]] The link in the <small>Database Connection</small> section will take you to the connection parameters dialog. Fill in the values for your database, press <small>Save</small> and the connection status should turn into a nice green <small>OK</small>. '''DBMS:''' Select the entry appropriate for your collection. The BPS can directly connect to database servers (MySQL, Postgres, DB2, SQL Server, and Oracle) and to Desktop-DBMS like Access and FoxPro through ODBC. ODBC is only available on Windows, so if you plan to use it, BioCASe must be installed on a Windows Server. '''Host:''' Enter the IP address of the database server. If you’re connecting through ODBC, you must leave this empty. Instead, use Windows’ ODBC Administrator to set up a System (!) datasource; the server or file name of the database will be stored in the ODBC configuration. '''Database:''' Enter the name of the ODBC datasource if you’re using ODBC, or the name of the database if you’re directly connecting to a server. '''User/Password:''' Database credentials to be used. As written in the [[Preparation]] guide, you should create a user with just read access for this. '''Encoding:''' The encoding of your database. If you’re not sure, leave the default, you can change it if you run into problems later. If you know your database is using a different encoding than latin-1 (which is typically the default for western languages), maybe UTF-8, select the appropriate entry in the drop-down list. [[File:dsaDBConnection.png|450px]] Once you’ve entered the correct values, press <small>Save</small>. Click <small>Overview</small> in the link list at the top of the page to return to the datasource configuration page. If the connection status doesn’t turn into <small>OK</small> (and doesn’t after you’ve returned to the overview page), please read the [[Debugging]] guide about how to trace the problem. == Setting up the Database Structure == Clicking on <small>Edit DB Structure</small> on the datasource overview page will show the DB structure setup. Make sure the connection status at the top of the page still is <small>OK</small>. In a first step, you will add all tables/views that hold information you want to publish with BioCASe. You can ignore all database tables storing only data you don’t plan to map to ABCD. Same holds true if you use cache tables (as explained in the [[Preparation]] guide): Only the cache tables need to be set up on the database structure page. In a second step, you will specify the foreign key references between these tables. === Adding Tables/Views with their Primary Keys === To add a table or view, just select it in the drop-down list. The name of the table will be filled in the <small>Alias</small> text box (you will use this alias later in the mapping process), feel free to change it into something that describes the contents of that tables (for example ''specimen'' for the root specimen table). Press <small>Add</small> to insert the table into the list. Before continuing to the next table, select the primary key in the list of attributes under <small>Primary Key Attribute(s)</small>. In case the data type is a string and not an integer, select <small>text</small> from the drop-down list. Press <small>Add</small> to add the primary key. If the primary key consists of several columns, you can add several primary key attributes. Redo these steps (adding the tables first, then the primary key/keys) for all tables you want to publish data from. Always remember to set the correct primary key column(s), even if they’re not defined as primary keys in the database. '''Tables without a primary key cannot be used by the Provider Software and will be removed upon saving!''' === Specifying Foreign Key References === Once you’ve added all tables, you can define the relations between these tables by defining foreign keys. In order to do this, go to the <small>Foreign Key(s)</small> column of the table that holds a foreign key reference to another table, select the alias of the referenced table and press <small>Add</small>. Select the attribute that stores the foreign key and change the data type if it is not integer, then press <small>Add</small>. You must specify as many attributes for the foreign keys as the primary key of the referenced table has, otherwise the Provider Software won’t be able to create a valid join between the tables. Also, they must be in the same order and have identical data types. Redo this for all tables until you’ve specified all foreign key references. If you’ve installed the Graphviz package, the Provider Software will create a graph visualizing the current database structure. Each table will be visible as a node with the primary key(s) listed. References between tables will be represented by arrows, the foreign key(s) listed in the referring table: [[File:dsaDatabaseStructure_AlgenEngels.png|800px]] === Saving, Editing an Existing Setup === You can change aliases of tables you’ve already added to the list. To do so, go to the <small>Alias</small> text box of that table and type in a different name. Upon exiting the text box, all references to this alias will be updated in the table setup; also the graph will reflect the changes. But caution: Schema mappings use these aliases to bind schema elements to tables and columns, if you change an alias you’ve already used in a schema mapping, you will break this binding! So it is advisable to decide for meaningful alias names and keep them stable once you’ve started the mapping process. Once you’ve added a table/view to the table list, you can’t change the underlying table, since this would invalidate all the primary and foreign key attributes. If you’ve added the wrong table, remove it by using the trash can symbol next to <small>Delete Alias</small> and add the correct table/view. If you’ve finished the table setup and the graph displays the model of your database correctly, save the configuration by pressing <small>Save</small> at the bottom of the page. This is important; navigating away from the page without saving will cause your changes to be lost (even though you will be asked for confirmation)! It is advisable to save the table setup every now and then, for example once you’ve added all tables and primary keys. If you edit an existing table setup and happen to realize you’ve messed it up, you can revert to the state when you’ve pressed Save for the last time by using the <small>Revert</small> button at the bottom. But be careful: All unsaved changes will be lost. '''<u>Important Note:</u>''' The retrieval of tables/view/column names is currently only supported for the following DBMS: Microsoft SQL Server, MySQL, Access, Postgres and FoxPro. If you use a different DBMS, you will see a simple text box instead of a drop-down-list, where you’ll have to type in the name of the table or column you want to add. cba99af25e09b8b5e194f7d8cdb4feeb920ebb40 330 329 2011-08-16T14:15:27Z JoergHoletschek 3 /* Saving, Editing an Existing Setup */ wikitext text/x-wiki This tutorial explains in detail how to set up a datasource. == A BioCASe Data Source == Publishing a database with BioCASe means it will be made accessible through a BioCASe web service, which will be used later by the biodiversity network(s) you want to join to access to your data. For example, the GBIF Harvester will send a BioCASe Scan request to that web service to get a list of all scientific names kept in the database, and subsequently several BioCASe Search requests to load all records for these names. For each database you want to publish, you need to set up a BioCASe data source (do not mix that up with ODBC data sources on Windows machines). The resulting BioCASe web service is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source. So if you made your installation available at http://www.foobar.org/biocase during the installation process and set up a data source named Herbar, the URL of the BioCASe web service would be http://www.foobar.org/biocase/pywrapper.cgi?dsa=Herbar (data source names can be case sensitive, depending on your server’s operating system). == Creating a New Data Source == The start page of the BPS will always list the data sources currently configured, that means the web services offered by that BioCASe installation. On a fresh installation, it will be empty: [[File:instBiocaseStartpage.png|500px]] Creating a new data source can be done in the configuration tool. Choose <small>Config Tool</small> on the start page, then <small>System Administration</small>. Log in with the password you set during the installation process (if you didn’t change that yet, the default is ''ACDC''). In the configuration screen, the sections <small>Data Sources</small> and <small>Recycle Bin</small> can be used to create, remove und restore a data source: [[File:dsaCreateDatasourceEmpty.png|500px]] For now, we want to create a new, empty data source. Enter a name – without any spaces – in the text box and click <small>Create DSA</small> (remember that the datasource name is case sensitive on Linux machines). After a second, it will appear in the list of existing data sources with a clickable link. By doing this, you created the first BioCASe web service for your installation. Clicking <small>Remove</small> will uninstall this service again (putting it in the recycle bin, so you can restore it later). Tip: If you select <small>abcdmetadata</small> in the template drop-down list, your data source will be created with an ABCD mapping for all the metadata elements stored in the metadata table mentioned in the [[Preparation]] guide. So if you did use the templates in ''sampledb/abcdmetadata'' to create the metadata table in your database, you can use this template to spare the hassle of mapping this table and start right away with your occurrence data. == Setting up the database connection == Clicking on the data source you’ve just created will take you to the datasource configuration tool. Note that the <small>Schemas</small> section will be empty if you’ve created an empty data source or list two schemas (ABCD 1.2 and 2.06) if you created the datasource based on the <small>abcdmetadata</small> template: [[File:dsaOverview.png|450px]] The link in the <small>Database Connection</small> section will take you to the connection parameters dialog. Fill in the values for your database, press <small>Save</small> and the connection status should turn into a nice green <small>OK</small>. '''DBMS:''' Select the entry appropriate for your collection. The BPS can directly connect to database servers (MySQL, Postgres, DB2, SQL Server, and Oracle) and to Desktop-DBMS like Access and FoxPro through ODBC. ODBC is only available on Windows, so if you plan to use it, BioCASe must be installed on a Windows Server. '''Host:''' Enter the IP address of the database server. If you’re connecting through ODBC, you must leave this empty. Instead, use Windows’ ODBC Administrator to set up a System (!) datasource; the server or file name of the database will be stored in the ODBC configuration. '''Database:''' Enter the name of the ODBC datasource if you’re using ODBC, or the name of the database if you’re directly connecting to a server. '''User/Password:''' Database credentials to be used. As written in the [[Preparation]] guide, you should create a user with just read access for this. '''Encoding:''' The encoding of your database. If you’re not sure, leave the default, you can change it if you run into problems later. If you know your database is using a different encoding than latin-1 (which is typically the default for western languages), maybe UTF-8, select the appropriate entry in the drop-down list. [[File:dsaDBConnection.png|450px]] Once you’ve entered the correct values, press <small>Save</small>. Click <small>Overview</small> in the link list at the top of the page to return to the datasource configuration page. If the connection status doesn’t turn into <small>OK</small> (and doesn’t after you’ve returned to the overview page), please read the [[Debugging]] guide about how to trace the problem. == Setting up the Database Structure == Clicking on <small>Edit DB Structure</small> on the datasource overview page will show the DB structure setup. Make sure the connection status at the top of the page still is <small>OK</small>. In a first step, you will add all tables/views that hold information you want to publish with BioCASe. You can ignore all database tables storing only data you don’t plan to map to ABCD. Same holds true if you use cache tables (as explained in the [[Preparation]] guide): Only the cache tables need to be set up on the database structure page. In a second step, you will specify the foreign key references between these tables. === Adding Tables/Views with their Primary Keys === To add a table or view, just select it in the drop-down list. The name of the table will be filled in the <small>Alias</small> text box (you will use this alias later in the mapping process), feel free to change it into something that describes the contents of that tables (for example ''specimen'' for the root specimen table). Press <small>Add</small> to insert the table into the list. Before continuing to the next table, select the primary key in the list of attributes under <small>Primary Key Attribute(s)</small>. In case the data type is a string and not an integer, select <small>text</small> from the drop-down list. Press <small>Add</small> to add the primary key. If the primary key consists of several columns, you can add several primary key attributes. Redo these steps (adding the tables first, then the primary key/keys) for all tables you want to publish data from. Always remember to set the correct primary key column(s), even if they’re not defined as primary keys in the database. '''Tables without a primary key cannot be used by the Provider Software and will be removed upon saving!''' === Specifying Foreign Key References === Once you’ve added all tables, you can define the relations between these tables by defining foreign keys. In order to do this, go to the <small>Foreign Key(s)</small> column of the table that holds a foreign key reference to another table, select the alias of the referenced table and press <small>Add</small>. Select the attribute that stores the foreign key and change the data type if it is not integer, then press <small>Add</small>. You must specify as many attributes for the foreign keys as the primary key of the referenced table has, otherwise the Provider Software won’t be able to create a valid join between the tables. Also, they must be in the same order and have identical data types. Redo this for all tables until you’ve specified all foreign key references. If you’ve installed the Graphviz package, the Provider Software will create a graph visualizing the current database structure. Each table will be visible as a node with the primary key(s) listed. References between tables will be represented by arrows, the foreign key(s) listed in the referring table: [[File:dsaDatabaseStructure_AlgenEngels.png|800px]] === Saving, Editing an Existing Setup === You can change aliases of tables you’ve already added to the list. To do so, go to the <small>Alias</small> text box of that table and type in a different name. Upon exiting the text box, all references to this alias will be updated in the table setup; also the graph will reflect the changes. But caution: Schema mappings use these aliases to bind schema elements to tables and columns, if you change an alias you’ve already used in a schema mapping, you will break this binding! So it is advisable to decide for meaningful alias names and keep them stable once you’ve started the mapping process. Once you’ve added a table/view to the table list, you can’t change the underlying table, since this would invalidate all the primary and foreign key attributes. If you’ve added the wrong table, remove it by using the trash can symbol next to <small>Delete Alias</small> and add the correct table/view. If you’ve finished the table setup and the graph displays the model of your database correctly, save the configuration by pressing <small>Save</small> at the bottom of the page. This is important; navigating away from the page without saving will cause your changes to be lost (even though you will be asked for confirmation)! It is advisable to save the table setup every now and then, for example once you’ve added all tables and primary keys. If you edit an existing table setup and happen to realize you’ve messed it up, you can revert to the state when you’ve pressed <small>Save</small> for the last time by using the <small>Revert</small> button at the bottom. But be careful: All unsaved changes will be lost. '''<u>Important Note:</u>''' The retrieval of tables/view/column names is currently only supported for the following DBMS: Microsoft SQL Server, MySQL, Access, Postgres and FoxPro. If you use a different DBMS, you will see a simple text box instead of a drop-down-list, where you’ll have to type in the name of the table or column you want to add. 71e861ecb6c8dad78bcc837690b2f3d3a60267c4 331 330 2011-08-16T14:16:22Z JoergHoletschek 3 /* Adding Tables/Views with their Primary Keys */ wikitext text/x-wiki This tutorial explains in detail how to set up a datasource. == A BioCASe Data Source == Publishing a database with BioCASe means it will be made accessible through a BioCASe web service, which will be used later by the biodiversity network(s) you want to join to access to your data. For example, the GBIF Harvester will send a BioCASe Scan request to that web service to get a list of all scientific names kept in the database, and subsequently several BioCASe Search requests to load all records for these names. For each database you want to publish, you need to set up a BioCASe data source (do not mix that up with ODBC data sources on Windows machines). The resulting BioCASe web service is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source. So if you made your installation available at http://www.foobar.org/biocase during the installation process and set up a data source named Herbar, the URL of the BioCASe web service would be http://www.foobar.org/biocase/pywrapper.cgi?dsa=Herbar (data source names can be case sensitive, depending on your server’s operating system). == Creating a New Data Source == The start page of the BPS will always list the data sources currently configured, that means the web services offered by that BioCASe installation. On a fresh installation, it will be empty: [[File:instBiocaseStartpage.png|500px]] Creating a new data source can be done in the configuration tool. Choose <small>Config Tool</small> on the start page, then <small>System Administration</small>. Log in with the password you set during the installation process (if you didn’t change that yet, the default is ''ACDC''). In the configuration screen, the sections <small>Data Sources</small> and <small>Recycle Bin</small> can be used to create, remove und restore a data source: [[File:dsaCreateDatasourceEmpty.png|500px]] For now, we want to create a new, empty data source. Enter a name – without any spaces – in the text box and click <small>Create DSA</small> (remember that the datasource name is case sensitive on Linux machines). After a second, it will appear in the list of existing data sources with a clickable link. By doing this, you created the first BioCASe web service for your installation. Clicking <small>Remove</small> will uninstall this service again (putting it in the recycle bin, so you can restore it later). Tip: If you select <small>abcdmetadata</small> in the template drop-down list, your data source will be created with an ABCD mapping for all the metadata elements stored in the metadata table mentioned in the [[Preparation]] guide. So if you did use the templates in ''sampledb/abcdmetadata'' to create the metadata table in your database, you can use this template to spare the hassle of mapping this table and start right away with your occurrence data. == Setting up the database connection == Clicking on the data source you’ve just created will take you to the datasource configuration tool. Note that the <small>Schemas</small> section will be empty if you’ve created an empty data source or list two schemas (ABCD 1.2 and 2.06) if you created the datasource based on the <small>abcdmetadata</small> template: [[File:dsaOverview.png|450px]] The link in the <small>Database Connection</small> section will take you to the connection parameters dialog. Fill in the values for your database, press <small>Save</small> and the connection status should turn into a nice green <small>OK</small>. '''DBMS:''' Select the entry appropriate for your collection. The BPS can directly connect to database servers (MySQL, Postgres, DB2, SQL Server, and Oracle) and to Desktop-DBMS like Access and FoxPro through ODBC. ODBC is only available on Windows, so if you plan to use it, BioCASe must be installed on a Windows Server. '''Host:''' Enter the IP address of the database server. If you’re connecting through ODBC, you must leave this empty. Instead, use Windows’ ODBC Administrator to set up a System (!) datasource; the server or file name of the database will be stored in the ODBC configuration. '''Database:''' Enter the name of the ODBC datasource if you’re using ODBC, or the name of the database if you’re directly connecting to a server. '''User/Password:''' Database credentials to be used. As written in the [[Preparation]] guide, you should create a user with just read access for this. '''Encoding:''' The encoding of your database. If you’re not sure, leave the default, you can change it if you run into problems later. If you know your database is using a different encoding than latin-1 (which is typically the default for western languages), maybe UTF-8, select the appropriate entry in the drop-down list. [[File:dsaDBConnection.png|450px]] Once you’ve entered the correct values, press <small>Save</small>. Click <small>Overview</small> in the link list at the top of the page to return to the datasource configuration page. If the connection status doesn’t turn into <small>OK</small> (and doesn’t after you’ve returned to the overview page), please read the [[Debugging]] guide about how to trace the problem. == Setting up the Database Structure == Clicking on <small>Edit DB Structure</small> on the datasource overview page will show the DB structure setup. Make sure the connection status at the top of the page still is <small>OK</small>. In a first step, you will add all tables/views that hold information you want to publish with BioCASe. You can ignore all database tables storing only data you don’t plan to map to ABCD. Same holds true if you use cache tables (as explained in the [[Preparation]] guide): Only the cache tables need to be set up on the database structure page. In a second step, you will specify the foreign key references between these tables. === Adding Tables/Views with their Primary Keys === To add a table or view, just select it in the drop-down list. The name of the table will be filled in the <small>Alias</small> text box (you will use this alias later in the mapping process), feel free to change it into something that describes the contents of that tables (for example ''specimen'' for the root specimen table). Press <small>Add</small> to insert the table into the list. Before continuing to the next table, select the primary key in the list of attributes under <small>Primary Key Attribute(s)</small>. In case the data type is a string and not an integer, select <small>text</small> from the drop-down list. Press <small>Add</small> to add the primary key. If the primary key consists of several columns, you can add several primary key attributes. Redo these steps (adding the tables first, then the primary key/keys) for all tables you want to publish data from. Always remember to set the correct primary key column(s), even if they’re not defined as primary keys in the database.<br> '''Important Note:''' Tables without a primary key cannot be used by the Provider Software and will be removed upon saving!''' === Specifying Foreign Key References === Once you’ve added all tables, you can define the relations between these tables by defining foreign keys. In order to do this, go to the <small>Foreign Key(s)</small> column of the table that holds a foreign key reference to another table, select the alias of the referenced table and press <small>Add</small>. Select the attribute that stores the foreign key and change the data type if it is not integer, then press <small>Add</small>. You must specify as many attributes for the foreign keys as the primary key of the referenced table has, otherwise the Provider Software won’t be able to create a valid join between the tables. Also, they must be in the same order and have identical data types. Redo this for all tables until you’ve specified all foreign key references. If you’ve installed the Graphviz package, the Provider Software will create a graph visualizing the current database structure. Each table will be visible as a node with the primary key(s) listed. References between tables will be represented by arrows, the foreign key(s) listed in the referring table: [[File:dsaDatabaseStructure_AlgenEngels.png|800px]] === Saving, Editing an Existing Setup === You can change aliases of tables you’ve already added to the list. To do so, go to the <small>Alias</small> text box of that table and type in a different name. Upon exiting the text box, all references to this alias will be updated in the table setup; also the graph will reflect the changes. But caution: Schema mappings use these aliases to bind schema elements to tables and columns, if you change an alias you’ve already used in a schema mapping, you will break this binding! So it is advisable to decide for meaningful alias names and keep them stable once you’ve started the mapping process. Once you’ve added a table/view to the table list, you can’t change the underlying table, since this would invalidate all the primary and foreign key attributes. If you’ve added the wrong table, remove it by using the trash can symbol next to <small>Delete Alias</small> and add the correct table/view. If you’ve finished the table setup and the graph displays the model of your database correctly, save the configuration by pressing <small>Save</small> at the bottom of the page. This is important; navigating away from the page without saving will cause your changes to be lost (even though you will be asked for confirmation)! It is advisable to save the table setup every now and then, for example once you’ve added all tables and primary keys. If you edit an existing table setup and happen to realize you’ve messed it up, you can revert to the state when you’ve pressed <small>Save</small> for the last time by using the <small>Revert</small> button at the bottom. But be careful: All unsaved changes will be lost. '''<u>Important Note:</u>''' The retrieval of tables/view/column names is currently only supported for the following DBMS: Microsoft SQL Server, MySQL, Access, Postgres and FoxPro. If you use a different DBMS, you will see a simple text box instead of a drop-down-list, where you’ll have to type in the name of the table or column you want to add. f1d2b67ae0c8cd49e5bdf0611b445c2229a87e8e 332 331 2011-08-16T14:16:49Z JoergHoletschek 3 /* Adding Tables/Views with their Primary Keys */ wikitext text/x-wiki This tutorial explains in detail how to set up a datasource. == A BioCASe Data Source == Publishing a database with BioCASe means it will be made accessible through a BioCASe web service, which will be used later by the biodiversity network(s) you want to join to access to your data. For example, the GBIF Harvester will send a BioCASe Scan request to that web service to get a list of all scientific names kept in the database, and subsequently several BioCASe Search requests to load all records for these names. For each database you want to publish, you need to set up a BioCASe data source (do not mix that up with ODBC data sources on Windows machines). The resulting BioCASe web service is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source. So if you made your installation available at http://www.foobar.org/biocase during the installation process and set up a data source named Herbar, the URL of the BioCASe web service would be http://www.foobar.org/biocase/pywrapper.cgi?dsa=Herbar (data source names can be case sensitive, depending on your server’s operating system). == Creating a New Data Source == The start page of the BPS will always list the data sources currently configured, that means the web services offered by that BioCASe installation. On a fresh installation, it will be empty: [[File:instBiocaseStartpage.png|500px]] Creating a new data source can be done in the configuration tool. Choose <small>Config Tool</small> on the start page, then <small>System Administration</small>. Log in with the password you set during the installation process (if you didn’t change that yet, the default is ''ACDC''). In the configuration screen, the sections <small>Data Sources</small> and <small>Recycle Bin</small> can be used to create, remove und restore a data source: [[File:dsaCreateDatasourceEmpty.png|500px]] For now, we want to create a new, empty data source. Enter a name – without any spaces – in the text box and click <small>Create DSA</small> (remember that the datasource name is case sensitive on Linux machines). After a second, it will appear in the list of existing data sources with a clickable link. By doing this, you created the first BioCASe web service for your installation. Clicking <small>Remove</small> will uninstall this service again (putting it in the recycle bin, so you can restore it later). Tip: If you select <small>abcdmetadata</small> in the template drop-down list, your data source will be created with an ABCD mapping for all the metadata elements stored in the metadata table mentioned in the [[Preparation]] guide. So if you did use the templates in ''sampledb/abcdmetadata'' to create the metadata table in your database, you can use this template to spare the hassle of mapping this table and start right away with your occurrence data. == Setting up the database connection == Clicking on the data source you’ve just created will take you to the datasource configuration tool. Note that the <small>Schemas</small> section will be empty if you’ve created an empty data source or list two schemas (ABCD 1.2 and 2.06) if you created the datasource based on the <small>abcdmetadata</small> template: [[File:dsaOverview.png|450px]] The link in the <small>Database Connection</small> section will take you to the connection parameters dialog. Fill in the values for your database, press <small>Save</small> and the connection status should turn into a nice green <small>OK</small>. '''DBMS:''' Select the entry appropriate for your collection. The BPS can directly connect to database servers (MySQL, Postgres, DB2, SQL Server, and Oracle) and to Desktop-DBMS like Access and FoxPro through ODBC. ODBC is only available on Windows, so if you plan to use it, BioCASe must be installed on a Windows Server. '''Host:''' Enter the IP address of the database server. If you’re connecting through ODBC, you must leave this empty. Instead, use Windows’ ODBC Administrator to set up a System (!) datasource; the server or file name of the database will be stored in the ODBC configuration. '''Database:''' Enter the name of the ODBC datasource if you’re using ODBC, or the name of the database if you’re directly connecting to a server. '''User/Password:''' Database credentials to be used. As written in the [[Preparation]] guide, you should create a user with just read access for this. '''Encoding:''' The encoding of your database. If you’re not sure, leave the default, you can change it if you run into problems later. If you know your database is using a different encoding than latin-1 (which is typically the default for western languages), maybe UTF-8, select the appropriate entry in the drop-down list. [[File:dsaDBConnection.png|450px]] Once you’ve entered the correct values, press <small>Save</small>. Click <small>Overview</small> in the link list at the top of the page to return to the datasource configuration page. If the connection status doesn’t turn into <small>OK</small> (and doesn’t after you’ve returned to the overview page), please read the [[Debugging]] guide about how to trace the problem. == Setting up the Database Structure == Clicking on <small>Edit DB Structure</small> on the datasource overview page will show the DB structure setup. Make sure the connection status at the top of the page still is <small>OK</small>. In a first step, you will add all tables/views that hold information you want to publish with BioCASe. You can ignore all database tables storing only data you don’t plan to map to ABCD. Same holds true if you use cache tables (as explained in the [[Preparation]] guide): Only the cache tables need to be set up on the database structure page. In a second step, you will specify the foreign key references between these tables. === Adding Tables/Views with their Primary Keys === To add a table or view, just select it in the drop-down list. The name of the table will be filled in the <small>Alias</small> text box (you will use this alias later in the mapping process), feel free to change it into something that describes the contents of that tables (for example ''specimen'' for the root specimen table). Press <small>Add</small> to insert the table into the list. Before continuing to the next table, select the primary key in the list of attributes under <small>Primary Key Attribute(s)</small>. In case the data type is a string and not an integer, select <small>text</small> from the drop-down list. Press <small>Add</small> to add the primary key. If the primary key consists of several columns, you can add several primary key attributes. Redo these steps (adding the tables first, then the primary key/keys) for all tables you want to publish data from. Always remember to set the correct primary key column(s), even if they’re not defined as primary keys in the database.<br> '''<u>Important Note:</u>''' Tables without a primary key cannot be used by the Provider Software and will be removed upon saving!''' === Specifying Foreign Key References === Once you’ve added all tables, you can define the relations between these tables by defining foreign keys. In order to do this, go to the <small>Foreign Key(s)</small> column of the table that holds a foreign key reference to another table, select the alias of the referenced table and press <small>Add</small>. Select the attribute that stores the foreign key and change the data type if it is not integer, then press <small>Add</small>. You must specify as many attributes for the foreign keys as the primary key of the referenced table has, otherwise the Provider Software won’t be able to create a valid join between the tables. Also, they must be in the same order and have identical data types. Redo this for all tables until you’ve specified all foreign key references. If you’ve installed the Graphviz package, the Provider Software will create a graph visualizing the current database structure. Each table will be visible as a node with the primary key(s) listed. References between tables will be represented by arrows, the foreign key(s) listed in the referring table: [[File:dsaDatabaseStructure_AlgenEngels.png|800px]] === Saving, Editing an Existing Setup === You can change aliases of tables you’ve already added to the list. To do so, go to the <small>Alias</small> text box of that table and type in a different name. Upon exiting the text box, all references to this alias will be updated in the table setup; also the graph will reflect the changes. But caution: Schema mappings use these aliases to bind schema elements to tables and columns, if you change an alias you’ve already used in a schema mapping, you will break this binding! So it is advisable to decide for meaningful alias names and keep them stable once you’ve started the mapping process. Once you’ve added a table/view to the table list, you can’t change the underlying table, since this would invalidate all the primary and foreign key attributes. If you’ve added the wrong table, remove it by using the trash can symbol next to <small>Delete Alias</small> and add the correct table/view. If you’ve finished the table setup and the graph displays the model of your database correctly, save the configuration by pressing <small>Save</small> at the bottom of the page. This is important; navigating away from the page without saving will cause your changes to be lost (even though you will be asked for confirmation)! It is advisable to save the table setup every now and then, for example once you’ve added all tables and primary keys. If you edit an existing table setup and happen to realize you’ve messed it up, you can revert to the state when you’ve pressed <small>Save</small> for the last time by using the <small>Revert</small> button at the bottom. But be careful: All unsaved changes will be lost. '''<u>Important Note:</u>''' The retrieval of tables/view/column names is currently only supported for the following DBMS: Microsoft SQL Server, MySQL, Access, Postgres and FoxPro. If you use a different DBMS, you will see a simple text box instead of a drop-down-list, where you’ll have to type in the name of the table or column you want to add. a34b839e59ab7b601296c35dcd1da72b75ad7961 0 48 321 121 2011-08-16T13:57:09Z JoergHoletschek 3 wikitext text/x-wiki ;ABCD Data Schema (Access to Biological Collection Data): XML data schema used for publishing specimen collections and observational databases to primary biodiversity networks such as BioCASe and GBIF. The current version of ABCD is 2.06. ;Apache HTTP Server (“Apache”): Widely used open-source web server software. Apache is available for most operating systems and recommended for running BioCASe. ;BioCASe Installation: An installation of the BioCASe Provider Software that offers one or several BioCASe Web Services. ;BioCASe Web Service: A web service offered by a BioCASe installation. A BioCASe web service provides access to the database to be published and can be queried using the BioCASe Protocol. It is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source, for example http://www.foobar.org/biocase/pywrapper.cgi?dsa=Herbar. See [[DatasourceSetup]] for more information on how to set up a BioCASe Web Service. ;BioCASe (Biological Collection Access Service) Network: A network of primary biodiversity data providers. Currently, BioCASe offers three different data portals: One on the German flora (http://search.biocase.de/botany), one for the biodiversity of Europe (http://search.biocase.org/europe) and one for global occurrence data (http://search.biocase.org/edit). ;BioCASE Project (Biological Collection Access Service for Europe): EU-funded project (2002-05) that developed standards and software for publishing primary biodiversity data (specimen collections and observational databases). Within the first years, the results comprised the BioCASe Protocol, the ABCD data schema and the BioCASe Provider Software. ;BioCASe Protocol: The protocol for querying a BioCASe web service. The protocol defines three types of requests: ''Capabilities'' for introspecting the configuration of a BioCASe web service, ''Scan'' for getting the distinctive values for a concept provided by the web service, and ''Search'' for retrieving the full set of information published for certain records. ;BioCASe Request: See BioCASe Protocol. ;BioCASe Technology (Biological Collection Access Service): The suite of standards and software developed by the BioCASE Project and its successors. Currently, the main components are the BioCASe Protocol, the ABCD data schema, BioCASe Provider Software, the SYNTHESYS Cache Generation System and the BioCASe Data Portal Software. ;BPS: BioCASe Provider Software; also called PyWrapper. ;Cache: A component (in the BioCASe context, usually a database or several database tables) that temporarily stores data from another source. In most cases, this is done in order to increase performance for information retrieval. ;Configuration tool (also Config Tool): Set of dialogs used for configuring a BioCASe installation or BioCASe web services. ;Database View: A virtual table in a database that offers another view on the data stored in the database. A view can combine data from several tables and transform, that is, translate, concatenate or pivot them. Updates of the underlying tables are always reflected in views drawing data from these tables. ;Datasource: In the BioCASe context, a datasource is database that is to be published as a BioCASe web service. Do not confuse that with an ODBC datasource that defines the connection parameters for connecting to a database through ODBC. ;DBMS: Database Management System. ;Debugging: The iterative process of testing, finding and correcting errors. ;Denormalisation: In a relational database, the information for real world objects are split into several tables, which is called normalisation. Denormalisation is the process of intentionally reversing this step for the sake of faster information retrieval. ;Firewall: Usually one device or a combination of several devices shielding a local area network (LAN) from the internet. A firewall filters all network traffic between the LAN and the Internet, blocking unwanted protocols (i.e. applications) and ports. In a BioCASe context, servers should be placed in a way that no firewall is located between the web server running the Provider Software and the database server. In case there is a firewall in between, the port used by the database management system (e.g. 3306 for MySQL) must be opened. ;Global Biodiversity Information Facility (GBIF): International organisation for promoting free and universal access to worldwide primary biodiversity information. One of the main services offered is the GBIF Data Portal (http://data.gbif.org). ;Harvester: Application that retrieves all records published by a (BioCASe) web service and stores them in a local database. Harvesters are run by projects and initiatives that consume or aggregate primary biodiversity information, for example by GBIF for running the [http://data.gbif.org GBIF data portal]. ;Identifier: In a database table, an identifier is a field that names a record in that table. Identifiers are usually unique, that means the identifier is different for each record. ;Internet Information Services (IIS): Web service platform integrated into Microsoft’s server operating systems (Server 2000, 2003 and 2008); can be used for running the BioCASe Provider Software. ;Local Query Tool: Component of the BioCASe Provider Software that can be used to access a BioCASe web service. The Query Tool offers a customizable, very simple and no-frills search interface for querying '''one''' (!) BioCASe web service. ;Mapping Editor: Page of the configuration tool for mapping concepts of the published data schema to the source tables/columns in the database. ;Metadata: Data about data. In the BioCASe context, this is a set of information describing the database published as a whole - title, description, curator’s name, ownership, date of last modification, terms of use etc. ;Open Database Connectivity (ODBC): Standard Interface for accessing relational database management systems. The connection parameters for connecting to a database through ODBC are defined in an ODBC Datasource. ;OS: Operating System (Windows, MacOS, Ubuntu, Debian, Ret Hat, Solaris etc.). ;Python: Platform independent programming language used for the BioCASe Provider Software. ;PyWrapper: Synonym for the BioCASe Provider Software (BPS), created from "Python Wrapper". Sort of "wraps" around the database to be published, exposing the data as a BioCASe compliant web service. ;Query Form: Component of the Provider Software for sending requests to a BioCASe web service and displaying the result documents; usually used for debugging a web service. ;Schema Mapping: The part of the configuration that tells the Provider Software how to create documents in a specific published data schema from the source database. A BioCASe web service can support several schemas (e.g. for ABCD and DarwinCore) by having several schema mappings. ;SQL (Structured Query Language): Language for managing data in relational database management systems, e.g. for information retrieval. ;Subversion: Software versioning and revision control system used for the BioCASe Provider Software. When installing BioCASe, the software can be downloaded directly from the Subversion repository (See [[Installation#Downloading from the Subversion Repository]]). ;Wrapper: In the BioCASe context a synonym for PyWrapper. 65d0625311af783538cbeb866c1b25214c9746da 322 321 2011-08-16T13:57:44Z JoergHoletschek 3 wikitext text/x-wiki ;ABCD Data Schema (Access to Biological Collection Data): XML data schema used for publishing specimen collections and observational databases to primary biodiversity networks such as BioCASe and GBIF. The current version of ABCD is 2.06. ;Apache HTTP Server (“Apache”): Widely used open-source web server software. Apache is available for most operating systems and recommended for running BioCASe. ;BioCASe Installation: An installation of the BioCASe Provider Software that offers one or several BioCASe Web Services. ;BioCASe Web Service: A web service offered by a BioCASe installation. A BioCASe web service provides access to the database to be published and can be queried using the BioCASe Protocol. It is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source, for example http://www.myinstitution.org/biocase/pywrapper.cgi?dsa=Herbar. See [[DatasourceSetup]] for more information on how to set up a BioCASe Web Service. ;BioCASe (Biological Collection Access Service) Network: A network of primary biodiversity data providers. Currently, BioCASe offers three different data portals: One on the German flora (http://search.biocase.de/botany), one for the biodiversity of Europe (http://search.biocase.org/europe) and one for global occurrence data (http://search.biocase.org/edit). ;BioCASE Project (Biological Collection Access Service for Europe): EU-funded project (2002-05) that developed standards and software for publishing primary biodiversity data (specimen collections and observational databases). Within the first years, the results comprised the BioCASe Protocol, the ABCD data schema and the BioCASe Provider Software. ;BioCASe Protocol: The protocol for querying a BioCASe web service. The protocol defines three types of requests: ''Capabilities'' for introspecting the configuration of a BioCASe web service, ''Scan'' for getting the distinctive values for a concept provided by the web service, and ''Search'' for retrieving the full set of information published for certain records. ;BioCASe Request: See BioCASe Protocol. ;BioCASe Technology (Biological Collection Access Service): The suite of standards and software developed by the BioCASE Project and its successors. Currently, the main components are the BioCASe Protocol, the ABCD data schema, BioCASe Provider Software, the SYNTHESYS Cache Generation System and the BioCASe Data Portal Software. ;BPS: BioCASe Provider Software; also called PyWrapper. ;Cache: A component (in the BioCASe context, usually a database or several database tables) that temporarily stores data from another source. In most cases, this is done in order to increase performance for information retrieval. ;Configuration tool (also Config Tool): Set of dialogs used for configuring a BioCASe installation or BioCASe web services. ;Database View: A virtual table in a database that offers another view on the data stored in the database. A view can combine data from several tables and transform, that is, translate, concatenate or pivot them. Updates of the underlying tables are always reflected in views drawing data from these tables. ;Datasource: In the BioCASe context, a datasource is database that is to be published as a BioCASe web service. Do not confuse that with an ODBC datasource that defines the connection parameters for connecting to a database through ODBC. ;DBMS: Database Management System. ;Debugging: The iterative process of testing, finding and correcting errors. ;Denormalisation: In a relational database, the information for real world objects are split into several tables, which is called normalisation. Denormalisation is the process of intentionally reversing this step for the sake of faster information retrieval. ;Firewall: Usually one device or a combination of several devices shielding a local area network (LAN) from the internet. A firewall filters all network traffic between the LAN and the Internet, blocking unwanted protocols (i.e. applications) and ports. In a BioCASe context, servers should be placed in a way that no firewall is located between the web server running the Provider Software and the database server. In case there is a firewall in between, the port used by the database management system (e.g. 3306 for MySQL) must be opened. ;Global Biodiversity Information Facility (GBIF): International organisation for promoting free and universal access to worldwide primary biodiversity information. One of the main services offered is the GBIF Data Portal (http://data.gbif.org). ;Harvester: Application that retrieves all records published by a (BioCASe) web service and stores them in a local database. Harvesters are run by projects and initiatives that consume or aggregate primary biodiversity information, for example by GBIF for running the [http://data.gbif.org GBIF data portal]. ;Identifier: In a database table, an identifier is a field that names a record in that table. Identifiers are usually unique, that means the identifier is different for each record. ;Internet Information Services (IIS): Web service platform integrated into Microsoft’s server operating systems (Server 2000, 2003 and 2008); can be used for running the BioCASe Provider Software. ;Local Query Tool: Component of the BioCASe Provider Software that can be used to access a BioCASe web service. The Query Tool offers a customizable, very simple and no-frills search interface for querying '''one''' (!) BioCASe web service. ;Mapping Editor: Page of the configuration tool for mapping concepts of the published data schema to the source tables/columns in the database. ;Metadata: Data about data. In the BioCASe context, this is a set of information describing the database published as a whole - title, description, curator’s name, ownership, date of last modification, terms of use etc. ;Open Database Connectivity (ODBC): Standard Interface for accessing relational database management systems. The connection parameters for connecting to a database through ODBC are defined in an ODBC Datasource. ;OS: Operating System (Windows, MacOS, Ubuntu, Debian, Ret Hat, Solaris etc.). ;Python: Platform independent programming language used for the BioCASe Provider Software. ;PyWrapper: Synonym for the BioCASe Provider Software (BPS), created from "Python Wrapper". Sort of "wraps" around the database to be published, exposing the data as a BioCASe compliant web service. ;Query Form: Component of the Provider Software for sending requests to a BioCASe web service and displaying the result documents; usually used for debugging a web service. ;Schema Mapping: The part of the configuration that tells the Provider Software how to create documents in a specific published data schema from the source database. A BioCASe web service can support several schemas (e.g. for ABCD and DarwinCore) by having several schema mappings. ;SQL (Structured Query Language): Language for managing data in relational database management systems, e.g. for information retrieval. ;Subversion: Software versioning and revision control system used for the BioCASe Provider Software. When installing BioCASe, the software can be downloaded directly from the Subversion repository (See [[Installation#Downloading from the Subversion Repository]]). ;Wrapper: In the BioCASe context a synonym for PyWrapper. 0df3f2644ede117b99cd525a94d954199f03ebf5 333 322 2011-08-16T14:20:36Z JoergHoletschek 3 wikitext text/x-wiki ;ABCD Data Schema (Access to Biological Collection Data): XML data schema used for publishing specimen collections and observational databases to primary biodiversity networks such as BioCASe and GBIF. The current version of ABCD is 2.06. ;Apache HTTP Server (“Apache”): Widely used open-source web server software. Apache is available for most operating systems and recommended for running BioCASe. ;BioCASe Installation: An installation of the BioCASe Provider Software that offers one or several BioCASe Web Services. ;BioCASe Web Service: A web service offered by a BioCASe installation. A BioCASe web service provides access to the database to be published and can be queried using the BioCASe Protocol. It is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source, for example http://www.myinstitution.org/biocase/pywrapper.cgi?dsa=Herbar. See [[DatasourceSetup]] for more information on how to set up a BioCASe Web Service. ;BioCASe (Biological Collection Access Service) Network: A network of primary biodiversity data providers. Currently, BioCASe offers three different data portals: One on the German flora (http://search.biocase.de/botany), one for the biodiversity of Europe (http://search.biocase.org/europe) and one for global occurrence data (http://search.biocase.org/edit). ;BioCASE Project (Biological Collection Access Service for Europe): EU-funded project (2002-05) that developed standards and software for publishing primary biodiversity data (specimen collections and observational databases). Within the first years, the results comprised the BioCASe Protocol, the ABCD data schema and the BioCASe Provider Software. ;BioCASe Protocol: The protocol for querying a BioCASe web service. The protocol defines three types of requests: ''Capabilities'' for introspecting the configuration of a BioCASe web service, ''Scan'' for getting the distinctive values for a concept provided by the web service, and ''Search'' for retrieving the full set of information published for certain records. ;BioCASe Request: See BioCASe Protocol. ;BioCASe Technology (Biological Collection Access Service): The suite of standards and software developed by the BioCASE Project and its successors. Currently, the main components are the BioCASe Protocol, the ABCD data schema, BioCASe Provider Software, the SYNTHESYS Cache Generation System and the BioCASe Data Portal Software. ;BPS: BioCASe Provider Software; also called PyWrapper. ;Cache: A component (in the BioCASe context, usually a database or several database tables) that temporarily stores data from another source. In most cases, this is done in order to increase performance for information retrieval. ;Configuration tool (also Config Tool): Set of dialogs used for configuring a BioCASe installation or BioCASe web services. ;Database View: A virtual table in a database that offers another view on the data stored in the database. A view can combine data from several tables and transform, that is, translate, concatenate or pivot them. Updates of the underlying tables are always reflected in views drawing data from these tables. ;Datasource: In the BioCASe context, a datasource is database that is to be published as a BioCASe web service. Do not confuse that with an ODBC datasource that defines the connection parameters for connecting to a database through ODBC. ;DBMS: Database Management System. ;Debugging: The iterative process of testing, finding and correcting errors. ;Denormalisation: In a relational database, the information for real world objects are split into several tables, which is called normalisation. Denormalisation is the process of intentionally reversing this step for the sake of faster information retrieval. ;Firewall: Usually one device, a combination of several devices or a software shielding a local area network (LAN) from the internet. A firewall filters all network traffic between the LAN and the Internet, blocking unwanted protocols (i.e. applications) and ports. In a BioCASe context, servers should be placed in a way that no firewall is located between the web server running the Provider Software and the database server. In case there is a firewall in between, the port used by the database management system (e.g. 3306 for MySQL) must be opened. ;Global Biodiversity Information Facility (GBIF): International organisation for promoting free and universal access to worldwide primary biodiversity information. One of the main services offered is the GBIF Data Portal (http://data.gbif.org). ;Harvester: Application that retrieves all records published by a (BioCASe) web service and stores them in a local database. Harvesters are run by projects and initiatives that consume or aggregate primary biodiversity information, for example by GBIF for running the [http://data.gbif.org GBIF data portal]. ;Identifier: In a database table, an identifier is a field that names a record in that table. Identifiers are usually unique, that means the identifier is different for each record. ;Internet Information Services (IIS): Web service platform integrated into Microsoft’s server operating systems (Server 2000, 2003 and 2008); can be used for running the BioCASe Provider Software. ;Local Query Tool: Component of the BioCASe Provider Software that can be used to access a BioCASe web service. The Query Tool offers a customizable, very simple and no-frills search interface for querying '''one''' (!) BioCASe web service. ;Mapping Editor: Page of the configuration tool for mapping concepts of the published data schema to the source tables/columns in the database. ;Metadata: Data about data. In the BioCASe context, this is a set of information describing the database published as a whole - title, description, curator’s name, ownership, date of last modification, terms of use etc. ;Open Database Connectivity (ODBC): Standard Interface for accessing relational database management systems. The connection parameters for connecting to a database through ODBC are defined in an ODBC Datasource. ;OS: Operating System (Windows, MacOS, Ubuntu, Debian, Ret Hat, Solaris etc.). ;Python: Platform independent programming language used for the BioCASe Provider Software. ;PyWrapper: Synonym for the BioCASe Provider Software (BPS), created from "Python Wrapper". Sort of "wraps" around the database to be published, exposing the data as a BioCASe compliant web service. ;Query Form: Component of the Provider Software for sending requests to a BioCASe web service and displaying the result documents; usually used for debugging a web service. ;Schema Mapping: The part of the configuration that tells the Provider Software how to create documents in a specific published data schema from the source database. A BioCASe web service can support several schemas (e.g. for ABCD and DarwinCore) by having several schema mappings. ;SQL (Structured Query Language): Language for managing data in relational database management systems, e.g. for information retrieval. ;Subversion: Software versioning and revision control system used for the BioCASe Provider Software. When installing BioCASe, the software can be downloaded directly from the Subversion repository (See [[Installation#Downloading from the Subversion Repository]]). ;Wrapper: In the BioCASe context a synonym for PyWrapper. b91e00327447e20b2c433aa3cc330941a319b118 0 52 334 296 2011-08-16T14:25:25Z JoergHoletschek 3 /* When can I use the BioCASe Provider Software? */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just contact us and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry (you might need to close and reopen your browser). If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I cannot find the installation package for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. == Trouble Shooting == ===I’ve forgot the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map it to the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. f1aab34967fa680aabe20a02441004526a1c0c02 335 334 2011-08-16T14:26:33Z JoergHoletschek 3 /* General */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry (you might need to close and reopen your browser). If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I cannot find the installation package for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. == Trouble Shooting == ===I’ve forgot the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map it to the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. 348782146da84dee318c5ea6abfb1a91a65e53eb 336 335 2011-08-16T14:28:51Z JoergHoletschek 3 /* When typing in the URL for my new installation (http://localhost/biocase), I get an empty page or a “URL not found” error. What’s wrong? */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I cannot find the installation package for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. == Trouble Shooting == ===I’ve forgot the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map it to the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. d64cdaaab6939a4a9af8df5d0678631e427959d4 337 336 2011-08-16T14:32:16Z JoergHoletschek 3 /* I cannot find the installation package for the DBMS and the Python version I am using. */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I cannot find the installation package of the database connectivity module for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. == Trouble Shooting == ===I’ve forgot the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map it to the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. 0db5581c6211b64f878d1020e021ae5087447705 338 337 2011-08-16T14:33:19Z JoergHoletschek 3 /* I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribut wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I cannot find the installation package of the database connectivity module for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later of psycopg2. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. == Trouble Shooting == ===I’ve forgot the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map it to the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. b728249092c9dfd8c88ed0a808462a58227e2f51 339 338 2011-08-16T14:36:10Z JoergHoletschek 3 /* I want to update my BioCASe installation. How do I do this? */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I cannot find the installation package of the database connectivity module for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later of psycopg2. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation to the ''config'' folder of your new installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. == Trouble Shooting == ===I’ve forgot the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map it to the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. 9b00bb2de78ce71fffdcc04fcdddcd156f7d9069 340 339 2011-08-16T14:44:40Z JoergHoletschek 3 /* I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong? */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I cannot find the installation package of the database connectivity module for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later of psycopg2. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation to the ''config'' folder of your new installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other only by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. == Trouble Shooting == ===I’ve forgot the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map it to the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. 629da45fe56be2cdf0115abe45c7a6a2f9877788 0 23 341 297 2011-08-17T10:09:19Z JoergHoletschek 3 /* Version 2.6.1 [upcoming] */ wikitext text/x-wiki == Version 2 History == ===Version 2.6.1 [upcoming]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.5.5 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. See example [http://ww3.bgbm.org/bps2/ScreenShots?action=AttachFile&do=get&target=config_tool4.gif here] * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 4357d402293633c9eb4c4bfb374bb6e420f0aefd 347 341 2011-08-18T08:50:53Z JoergHoletschek 3 /* Version 2.6.1 [upcoming] */ wikitext text/x-wiki == Version 2 History == ===Version 2.6.1 [upcoming]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler: Unary operator (isNull, isNotNull) are processed correctly now. ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.5.5 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. See example [http://ww3.bgbm.org/bps2/ScreenShots?action=AttachFile&do=get&target=config_tool4.gif here] * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 24c8ad5d13db1828cd893194a73874ccf4334fef 6 10 342 21 2011-08-17T10:24:54Z JoergHoletschek 3 uploaded a new version of &quot;[[File:InstSetupScriptStart.png]]&quot; wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 0 22 344 57 2011-08-17T10:33:40Z JoergHoletschek 3 /* Mandatory */ wikitext text/x-wiki This page lists the most commonly used ABCD 2.06 concepts. The ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a full list of ABCD concepts [http://wiki.tdwg.org/twiki/bin/view/ABCD/AbcdConcepts here]. == Mandatory == List of mandatory concepts. Without them being mapped the Provider Software will not be able to create valid ABCD document: Metadata * /DataSets/DataSet/ContentContacts/ContentContact/Name * /DataSets/DataSet/TechnicalContacts/TechnicalContact/Name * /DataSets/DataSet/Metadata/Description/Representation/Title * /DataSets/DataSet/Metadata/Description/Representation[@language] * /DataSets/DataSet/Metadata/RevisionData/DateModified Unitdata <pre> /DataSets/DataSet/Units/Unit/ ├─ SourceInstitutionID ├─ SourceID └─ UnitID </pre> == Highly Recommended == List of recommended ABCD concepts. If you can provide this data, please do so, they're commonly used entry points to your data. Metadata * /DataSets/DataSet/ContentContacts/ContentContact/Email * /DataSets/DataSet/ContentContacts/ContentContact/Address * /DataSets/DataSet/TechnicalContacts/TechnicalContact/Email * /DataSets/DataSet/TechnicalContacts/TechnicalContact/Address * /DataSets/DataSet/Metadata/Description/Representation/Details * /DataSets/DataSet/Metadata/Owners/Owner/Organisation/Name/Representation/Text * /DataSets/DataSet/Metadata/Owners/Owner/Person/FullName * /DataSets/DataSet/Metadata/Owners/Owner/URIs/URL * /DataSets/DataSet/Metadata/Owners/Owner/LogoURI Unitdata * /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString * /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/HigherTaxa/HigherTaxon/HigherTaxonName * /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/HigherTaxa/HigherTaxon/HigherTaxonRank * /DataSets/DataSet/Units/Unit/Identifications/Identification/PreferredFlag * /DataSets/DataSet/Units/Unit/RecordBasis * /DataSets/DataSet/Units/Unit/Gathering/DateTime/ISODateTimeBegin * /DataSets/DataSet/Units/Unit/Gathering/Agents/GatheringAgent/Person/FullName * /DataSets/DataSet/Units/Unit/Gathering/LocalityText * /DataSets/DataSet/Units/Unit/Gathering/Country/ISO3166Code * /DataSets/DataSet/Units/Unit/Gathering/Country/Name * /DataSets/DataSet/Units/Unit/Gathering/SiteCoordinateSets/SiteCoordinates/CoordinatesLatLong/LongitudeDecimal */DataSets/DataSet/Units/Unit/Gathering/SiteCoordinateSets/SiteCoordinates/CoordinatesLatLong/LatitudeDecimal == Commonly Used == List of further commonly mapped ABCD concepts. Metadata * /DataSets/DataSet/IPRStatements/TermsOfUseStatements/TermsOfUse/Text * /DataSets/DataSet/IPRStatements/Copyrights/Copyright/Text * /DataSets/DataSet/IPRStatements/IPRDeclarations/IPRDeclaration/Text * /DataSets/DataSet/IPRStatements/Licenses/License/Text * /DataSets/DataSet/IPRStatements/Disclaimers/Disclaimer/Text * /DataSets/DataSet/IPRStatements/Citations/Citation/Text * /DataSets/DataSet/IPRStatements/Acknowledgements/Acknowledgement/Text Unitdata * /DataSets/DataSet/Units/Unit/DateLastEdited * /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/InformalNameString * /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/InformalNameString[@language] * /DataSets/DataSet/Units/Unit/Identifications/Identification/Identifiers/Identifier/IdentifierPersonName/FullName * /DataSets/DataSet/Units/Unit/Identifications/Identification/Date/ISODateTimeBegin * /DataSets/DataSet/Units/Unit/KindOfUnit * /DataSets/DataSet/Units/Unit/SpecimenUnit/NomenclaturalTypeDesignations/NomenclaturalTypeDesignation/TypifiedName/FullScientificNameString * /DataSets/DataSet/Units/Unit/SpecimenUnit/NomenclaturalTypeDesignations/NomenclaturalTypeDesignation/TypeStatus */DataSets/DataSet/Units/Unit/SpecimenUnit/NomenclaturalTypeDesignations/NomenclaturalTypeDesignation/NomenclaturalReference/TitleCitation * /DataSets/DataSet/Units/Unit/SpecimenUnit/NomenclaturalTypeDesignations/NomenclaturalTypeDesignation/NomenclaturalReference/CitationDetail * /DataSets/DataSet/Units/Unit/MultiMediaObjects/MultiMediaObject/FileURI * /DataSets/DataSet/Units/Unit/MultiMediaObjects/MultiMediaObject/Comment * /DataSets/DataSet/Units/Unit/MultiMediaObjects/MultiMediaObject/Context * /DataSets/DataSet/Units/Unit/Gathering/Project/ProjectTitle * /DataSets/DataSet/Units/Unit/Gathering/NamedAreas/NamedArea/AreaName * /DataSets/DataSet/Units/Unit/Gathering/NamedAreas/NamedArea/AreaClass * /DataSets/DataSet/Units/Unit/Gathering/Altitude/MeasurementOrFactAtomised/LowerValue * /DataSets/DataSet/Units/Unit/Gathering/Altitude/MeasurementOrFactAtomised/UnitOfMeasurement * /DataSets/DataSet/Units/Unit/Gathering/Depth/MeasurementOrFactAtomised/LowerValue * /DataSets/DataSet/Units/Unit/Gathering/Depth/MeasurementOrFactAtomised/UnitOfMeasurement * /DataSets/DataSet/Units/Unit/Gathering/Biotope/Name * /DataSets/DataSet/Units/Unit/CollectorsFieldNumber * /DataSets/DataSet/Units/Unit/Sex * /DataSets/DataSet/Units/Unit/Age * /DataSets/DataSet/Units/Unit/Notes * /DataSets/DataSet/Units/Unit/RecordURI 568af446dc80f9a7611def6c81f44c2c01df81db 345 344 2011-08-17T10:39:47Z JoergHoletschek 3 /* Mandatory */ wikitext text/x-wiki This page lists the most commonly used ABCD 2.06 concepts. The ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a full list of ABCD concepts [http://wiki.tdwg.org/twiki/bin/view/ABCD/AbcdConcepts here]. == Mandatory == List of mandatory concepts. Without them being mapped the Provider Software will not be able to create valid ABCD document: '''Metadata''' <pre> DataSets/DataSet ├─ ContentContacts/ContentContact/Name ├─ TechnicalContacts/TechnicalContact/Name └─ Metadata ├─ Description │ ├─ Representation/Title │ └─ Representation[@language] └─ RevisionData/DateModified </pre> '''Unitdata''' <pre> /DataSets/DataSet/Units/Unit/ ├─ SourceInstitutionID ├─ SourceID └─ UnitID </pre> == Highly Recommended == List of recommended ABCD concepts. If you can provide this data, please do so, they're commonly used entry points to your data. Metadata * /DataSets/DataSet/ContentContacts/ContentContact/Email * /DataSets/DataSet/ContentContacts/ContentContact/Address * /DataSets/DataSet/TechnicalContacts/TechnicalContact/Email * /DataSets/DataSet/TechnicalContacts/TechnicalContact/Address * /DataSets/DataSet/Metadata/Description/Representation/Details * /DataSets/DataSet/Metadata/Owners/Owner/Organisation/Name/Representation/Text * /DataSets/DataSet/Metadata/Owners/Owner/Person/FullName * /DataSets/DataSet/Metadata/Owners/Owner/URIs/URL * /DataSets/DataSet/Metadata/Owners/Owner/LogoURI Unitdata * /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString * /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/HigherTaxa/HigherTaxon/HigherTaxonName * /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/HigherTaxa/HigherTaxon/HigherTaxonRank * /DataSets/DataSet/Units/Unit/Identifications/Identification/PreferredFlag * /DataSets/DataSet/Units/Unit/RecordBasis * /DataSets/DataSet/Units/Unit/Gathering/DateTime/ISODateTimeBegin * /DataSets/DataSet/Units/Unit/Gathering/Agents/GatheringAgent/Person/FullName * /DataSets/DataSet/Units/Unit/Gathering/LocalityText * /DataSets/DataSet/Units/Unit/Gathering/Country/ISO3166Code * /DataSets/DataSet/Units/Unit/Gathering/Country/Name * /DataSets/DataSet/Units/Unit/Gathering/SiteCoordinateSets/SiteCoordinates/CoordinatesLatLong/LongitudeDecimal */DataSets/DataSet/Units/Unit/Gathering/SiteCoordinateSets/SiteCoordinates/CoordinatesLatLong/LatitudeDecimal == Commonly Used == List of further commonly mapped ABCD concepts. Metadata * /DataSets/DataSet/IPRStatements/TermsOfUseStatements/TermsOfUse/Text * /DataSets/DataSet/IPRStatements/Copyrights/Copyright/Text * /DataSets/DataSet/IPRStatements/IPRDeclarations/IPRDeclaration/Text * /DataSets/DataSet/IPRStatements/Licenses/License/Text * /DataSets/DataSet/IPRStatements/Disclaimers/Disclaimer/Text * /DataSets/DataSet/IPRStatements/Citations/Citation/Text * /DataSets/DataSet/IPRStatements/Acknowledgements/Acknowledgement/Text Unitdata * /DataSets/DataSet/Units/Unit/DateLastEdited * /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/InformalNameString * /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/InformalNameString[@language] * /DataSets/DataSet/Units/Unit/Identifications/Identification/Identifiers/Identifier/IdentifierPersonName/FullName * /DataSets/DataSet/Units/Unit/Identifications/Identification/Date/ISODateTimeBegin * /DataSets/DataSet/Units/Unit/KindOfUnit * /DataSets/DataSet/Units/Unit/SpecimenUnit/NomenclaturalTypeDesignations/NomenclaturalTypeDesignation/TypifiedName/FullScientificNameString * /DataSets/DataSet/Units/Unit/SpecimenUnit/NomenclaturalTypeDesignations/NomenclaturalTypeDesignation/TypeStatus */DataSets/DataSet/Units/Unit/SpecimenUnit/NomenclaturalTypeDesignations/NomenclaturalTypeDesignation/NomenclaturalReference/TitleCitation * /DataSets/DataSet/Units/Unit/SpecimenUnit/NomenclaturalTypeDesignations/NomenclaturalTypeDesignation/NomenclaturalReference/CitationDetail * /DataSets/DataSet/Units/Unit/MultiMediaObjects/MultiMediaObject/FileURI * /DataSets/DataSet/Units/Unit/MultiMediaObjects/MultiMediaObject/Comment * /DataSets/DataSet/Units/Unit/MultiMediaObjects/MultiMediaObject/Context * /DataSets/DataSet/Units/Unit/Gathering/Project/ProjectTitle * /DataSets/DataSet/Units/Unit/Gathering/NamedAreas/NamedArea/AreaName * /DataSets/DataSet/Units/Unit/Gathering/NamedAreas/NamedArea/AreaClass * /DataSets/DataSet/Units/Unit/Gathering/Altitude/MeasurementOrFactAtomised/LowerValue * /DataSets/DataSet/Units/Unit/Gathering/Altitude/MeasurementOrFactAtomised/UnitOfMeasurement * /DataSets/DataSet/Units/Unit/Gathering/Depth/MeasurementOrFactAtomised/LowerValue * /DataSets/DataSet/Units/Unit/Gathering/Depth/MeasurementOrFactAtomised/UnitOfMeasurement * /DataSets/DataSet/Units/Unit/Gathering/Biotope/Name * /DataSets/DataSet/Units/Unit/CollectorsFieldNumber * /DataSets/DataSet/Units/Unit/Sex * /DataSets/DataSet/Units/Unit/Age * /DataSets/DataSet/Units/Unit/Notes * /DataSets/DataSet/Units/Unit/RecordURI 1d23540cfbda81fec25fa613b511464071e5a2e2 346 345 2011-08-17T10:48:24Z JoergHoletschek 3 /* Mandatory */ wikitext text/x-wiki This page lists the most commonly used ABCD 2.06 concepts. The ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a full list of ABCD concepts [http://wiki.tdwg.org/twiki/bin/view/ABCD/AbcdConcepts here]. == Mandatory == List of mandatory concepts. Without them being mapped the Provider Software will not be able to create valid ABCD document: <pre> DataSets └─ DataSet ├─ ContentContacts │ └─ ContentContact │ └─ Name ├─ TechnicalContacts │ └─ TechnicalContact │ └─ Name ├─ Metadata │ ├─ Description │ │ └─ Representation │ │ ├─ @language │ │ └─ Title │ └─ RevisionData │ └─ DateModified └─ Units └─ Unit ├─ SourceInstitutionID ├─ SourceID └─ UnitID </pre> == Highly Recommended == List of recommended ABCD concepts. If you can provide this data, please do so, they're commonly used entry points to your data. Metadata * /DataSets/DataSet/ContentContacts/ContentContact/Email * /DataSets/DataSet/ContentContacts/ContentContact/Address * /DataSets/DataSet/TechnicalContacts/TechnicalContact/Email * /DataSets/DataSet/TechnicalContacts/TechnicalContact/Address * /DataSets/DataSet/Metadata/Description/Representation/Details * /DataSets/DataSet/Metadata/Owners/Owner/Organisation/Name/Representation/Text * /DataSets/DataSet/Metadata/Owners/Owner/Person/FullName * /DataSets/DataSet/Metadata/Owners/Owner/URIs/URL * /DataSets/DataSet/Metadata/Owners/Owner/LogoURI Unitdata * /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString * /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/HigherTaxa/HigherTaxon/HigherTaxonName * /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/HigherTaxa/HigherTaxon/HigherTaxonRank * /DataSets/DataSet/Units/Unit/Identifications/Identification/PreferredFlag * /DataSets/DataSet/Units/Unit/RecordBasis * /DataSets/DataSet/Units/Unit/Gathering/DateTime/ISODateTimeBegin * /DataSets/DataSet/Units/Unit/Gathering/Agents/GatheringAgent/Person/FullName * /DataSets/DataSet/Units/Unit/Gathering/LocalityText * /DataSets/DataSet/Units/Unit/Gathering/Country/ISO3166Code * /DataSets/DataSet/Units/Unit/Gathering/Country/Name * /DataSets/DataSet/Units/Unit/Gathering/SiteCoordinateSets/SiteCoordinates/CoordinatesLatLong/LongitudeDecimal */DataSets/DataSet/Units/Unit/Gathering/SiteCoordinateSets/SiteCoordinates/CoordinatesLatLong/LatitudeDecimal == Commonly Used == List of further commonly mapped ABCD concepts. Metadata * /DataSets/DataSet/IPRStatements/TermsOfUseStatements/TermsOfUse/Text * /DataSets/DataSet/IPRStatements/Copyrights/Copyright/Text * /DataSets/DataSet/IPRStatements/IPRDeclarations/IPRDeclaration/Text * /DataSets/DataSet/IPRStatements/Licenses/License/Text * /DataSets/DataSet/IPRStatements/Disclaimers/Disclaimer/Text * /DataSets/DataSet/IPRStatements/Citations/Citation/Text * /DataSets/DataSet/IPRStatements/Acknowledgements/Acknowledgement/Text Unitdata * /DataSets/DataSet/Units/Unit/DateLastEdited * /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/InformalNameString * /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/InformalNameString[@language] * /DataSets/DataSet/Units/Unit/Identifications/Identification/Identifiers/Identifier/IdentifierPersonName/FullName * /DataSets/DataSet/Units/Unit/Identifications/Identification/Date/ISODateTimeBegin * /DataSets/DataSet/Units/Unit/KindOfUnit * /DataSets/DataSet/Units/Unit/SpecimenUnit/NomenclaturalTypeDesignations/NomenclaturalTypeDesignation/TypifiedName/FullScientificNameString * /DataSets/DataSet/Units/Unit/SpecimenUnit/NomenclaturalTypeDesignations/NomenclaturalTypeDesignation/TypeStatus */DataSets/DataSet/Units/Unit/SpecimenUnit/NomenclaturalTypeDesignations/NomenclaturalTypeDesignation/NomenclaturalReference/TitleCitation * /DataSets/DataSet/Units/Unit/SpecimenUnit/NomenclaturalTypeDesignations/NomenclaturalTypeDesignation/NomenclaturalReference/CitationDetail * /DataSets/DataSet/Units/Unit/MultiMediaObjects/MultiMediaObject/FileURI * /DataSets/DataSet/Units/Unit/MultiMediaObjects/MultiMediaObject/Comment * /DataSets/DataSet/Units/Unit/MultiMediaObjects/MultiMediaObject/Context * /DataSets/DataSet/Units/Unit/Gathering/Project/ProjectTitle * /DataSets/DataSet/Units/Unit/Gathering/NamedAreas/NamedArea/AreaName * /DataSets/DataSet/Units/Unit/Gathering/NamedAreas/NamedArea/AreaClass * /DataSets/DataSet/Units/Unit/Gathering/Altitude/MeasurementOrFactAtomised/LowerValue * /DataSets/DataSet/Units/Unit/Gathering/Altitude/MeasurementOrFactAtomised/UnitOfMeasurement * /DataSets/DataSet/Units/Unit/Gathering/Depth/MeasurementOrFactAtomised/LowerValue * /DataSets/DataSet/Units/Unit/Gathering/Depth/MeasurementOrFactAtomised/UnitOfMeasurement * /DataSets/DataSet/Units/Unit/Gathering/Biotope/Name * /DataSets/DataSet/Units/Unit/CollectorsFieldNumber * /DataSets/DataSet/Units/Unit/Sex * /DataSets/DataSet/Units/Unit/Age * /DataSets/DataSet/Units/Unit/Notes * /DataSets/DataSet/Units/Unit/RecordURI 6b0f7412261a9635e76cc7e79f8d88cc5a518206 2 74 348 2011-08-18T08:57:40Z DavidFichtmueller 4 show toolbox css text/css /* show toolbox */ #p-tb{ display:block;} 4b5ce491b204d5dbcc6d6f8dcd920d3f31a5ed0a 0 29 349 269 2011-08-18T11:44:55Z JoergHoletschek 3 /* Access to Biological Collection Data Schema (ABCD) */ wikitext text/x-wiki This tutorial will explain how to set up a mapping for the ABCD2.06 data schema. If you intend to map another schema (e.g. one of the ABCD extension ABCD-EFG, ABCD-DNA or HISPID), you still should take the time to read it. Large portions of the process will be identical. == Access to Biological Collection Data Schema (ABCD) == ABCD in its current version 2.06 is the data schema typically used in conjunction with BioCASe. It is a highly complex XML data schema with about 1,000 elements that is able to store almost every piece of information that can be found in a natural history specimen collection or an observation database. The complexity is due to the fact that it can be used for a wide range of collections/databases – for living and preserved specimens, observations and culture collections, for zoological, botanical, bacterial and viral collections, marine or terrestrial, for herbaria, botanic and zoological gardens. For each of these special types, ABCD features special sections where information specific for this type can be stored. Other sections of ABCD will be shared by all these types, for example for gathering site, identifier or metadata. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a list of commonly used ABCD elements on the Wiki page [[CommonABCD2Concepts|Common ABCD2 Concepts]]. == Creating an ABCD Mapping for a Data Source == If you created a data source from a template (e.g. ''abcdmetadata''), a schema mapping for ABCD will be already existent. In this case, you can directly jump to the next section. If you created an empty data source, the section <small>Schemas</small> in the data source configuration overview will be empty: [[File:abcdSchemaMappings.png]] In order to create a new schema mapping, select the desired schema from the list and click <small>Create</small>. For ABCD, choose <small>ABCD_2.06.xml</small> (be careful not to hit one of the ABCD extensions, they look quite similar). After clicking <small>Create</small>, you will be directed to the mapping editor, which will allow you to edit the mappings. == The Mapping Editor == The mapping editor allows you to add, remove and edit ABCD mappings. On top you’ll find some summary information, on the bottom you’ll see the structure of the ABCD mapping: [[File:abcdMappingEditorEmpty.png]] First thing we will do is to set the root table, which is the table that holds the records you want to publish (specimens or observations). That table must hold one record per object to be published, identified by a unique identifier (called ''UnitID''). This identifier doesn't need to be the primary key necessarily, as long as it is unique (if that sentence confuses you, ignore it and just keep in mind that the UnitID must be unique.) Use the drop down box labelled <small>Root table Alias</small> to select the root table of your data model. If your metadata table holds just one record (because all occurrences published share the same metadata) and if it is not linked to the root table by a foreign key, you can specify this table in the drop-down box <small>Static table Alias</small>. The Provider Software will use a natural join to connect the metadata record stored in this table to all occurrences published. If your metadata table is referenced by a foreign key in the root able, you must not declare the metadata table as a static table! So either join the metadata table using a foreign key OR declare it as a static table. When you’ve set the root table (and static table, if you use that), press <small>Save</small>. As with the DB structure setup, this will write your changes to the configuration files, closing the tab without saving will discard your changes! Also, in case you messed something up, you can use the <small>Revert</small> button to restore the state when you’ve saved for the last time. In the mapping editor, the link <small>Overview</small> at the top of the page will take you back to the datasource configuration overview. After creating a mapping, it will appear in the section <small>Schemas</small>. The entry will show the number of mapped elements (0 for now) and the schema namespace. Pressing the trash can symbol will move it to the trash can, which means that your web service does not support this schema anymore; if you did this by mistake, you can restore it by pressing the <small>Restore</small> button. <small>Purge</small> will delete the schema mapping permanently: [[File:abcdSchemaMappings2.png]] You can map several schemas for a given datasource. For example, in addition to the current ABCD2 standard, you could add the deprecated version 1.2 (even though I have no idea why you should do that) or DarwinCore. Just select the appropriate schema in the drop-down list and click <small>Create</small>. Since the namespace of a schema will be used in requests, you cannot create two mappings for the same namespace. Trying this will get you an error message. == Adding Mandatory ABCD Elements == Clicking on a schema in the datasource overview page will take you again to the mapping editor. In the lower half of the page, you will see the basic structure of an ABCD document: [[File:abcdMappingEditorEmptyBottom.png]] <strong>Technical/Content Contact:</strong> These trees hold information on who should be contacted for technical issues or questions concerning the published database, each with name, address, email address and telephone number. <strong>Metadata:</strong> In this sub tree you’ll find all metadata fields, for example title and description of the dataset, ownership information, date last modified, IPR statements etc. <strong>Units:</strong> This part stores the "real" data you want to publish, that means all specimen or observation data. The three elements already shown in the tree are identifiers that make up the so called "Triple ID", which uniquely identifies records within primary biodiversity networks such as BioCASe and GBIF: * ''UnitID'', which is a unique identifier for each record to be published (you can use the catalogue number, for example, or an existing barcode); * ''SourceInstitutionID'', an identifier for the institution publishing the dataset (for example BGBM); * ''SourceID'', an identifier for the dataset (e.g. “Herbarium Berolinense’’). The elements already shown in the mapping editor are printed in red because they are mandatory, which means they are required for an ABCD document. Without them being mapped (to database fields that are non-empty), the Provider Software will not be able to construct valid ABCD documents and will not publish any records. So the first elements that should be mapped are the mandatory fields. To add a mapping, click the <strong>+</strong> symbol next to the field, which will open the mapping editor dialog, where you can select the database table/column you want to map the ABCD element to: [[File:abcdMappingEditorDialog.png]] In this case, the element ''/DataSets/DataSet/Metadata/RevisionData/DateModified'' is mapped to the column ''source_update'' of the ''metadata'' table, which has the data type ''date''. (Remember that table/column retrieval is not supported for all DBMS, so if you’re unlucky, you might need to type in table/column name manually.) <small>Save</small> will accept your settings and close the dialog, <small>Cancel</small> will discard any changes. Instead of mapping an element to a database field, you can map it to a literal. For example, you could enter the technical and content contact information directly as literals in the mapping editor dialog (assuming they’re the same for all records). However, we strongly discourage you from doing this. For one thing, this is just bad style, since it will scatter the information published over different places: Some will be retrieved from the database, some (the literals) will be loaded from the configuration files. Changing the metadata of your service later would then require access to the database and access to the BioCASe configuration – that means the configuration tool password. Be nice to the person who will do these changes and put everything neatly in a database table! The second reason is even more important: Using literals for an element will result in this element to become not searchable, which means it cannot be used in a filter of a Search or a Scan request. Even though this behaviour is discoverable with a Capabilities request (by evaluating the ''searchable'' flag), a potential user of your web service might just not think of that, so it’s a good idea to avoid that in first place. Please use literals sparsely! In the mapping editor window, you can select several source columns and several literals, even a combination of both. The Provider Software will simply concatenate the values retrieved from the database and the literals specified (in the order they’re given). But again: Keep in mind that this will make the element not searchable. A better solution is to do any concatenation operations in a database view. Use the mapping editor window to add mappings for all eight mandatory fields: Click on the <strong>+</strong> symbol next to the element, choose table/column name and the correct data type, then press <small>Save</small>. When you’re done, the page should look similar to the following image. Press <small>Save</small> in the main window to write your changes to the configuration files. [[File:abcdMappingEditorMandatoryElements.png]] == Initial Testing of the Data Source == Once you’ve mapped all mandatory elements, you can do the first test of your BioCASe web service. Even though you haven’t mapped any real information yet, it is a good idea to check the service and solve any problems hitherto before continuing with the mapping. Click on <small>Test mapping!</small> to open the Query Form that can be used for this purpose: [[File:abcdQueryFormEmpty.png]] With the Query Form you can send a BioCASe request to your web service; it will wait for the reply of the web service and display the response document. On top of the page you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ''flora'' being the name of the datasource. Make sure this is correct for your installation! Into the text box below you can enter the request XML text. Because it would be cumbersome to enter the whole BioCASe request, you can use the links below the box to load templates. For now, click on <small>ABCD2 Search</small> to fill the box with a BioCASe Search request for documents in the ABCD2 format (the one we’ve just mapped): <syntaxhighlight> <?xml version='1.0' encoding='UTF-8'?> <request xmlns='http://www.biocase.org/schemas/protocol/1.3'> <header><type>search</type></header> <search> <requestFormat>http://www.tdwg.org/schemas/abcd/2.06</requestFormat> <responseFormat start='0' limit='10'>http://www.tdwg.org/schemas/abcd/2.06</responseFormat> <filter> <like path='/DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString'>A*</like> </filter> <count>false</count> </search> </request> </syntaxhighlight> As you can see, there is a filter set on the ''ScientificName'' element. Since we haven’t mapped this element yet, this won’t work. So let’s remove the filter (all lines between <filter> and </filter>), leaving the text box like this: [[File:abcdQueryFormFilled.png]] Pressing <small>Submit</small> will send the request to the web service and – hopefully – display the response document with the correct ABCD records. For the mapping we’ve just created, it should look similar to the following document (use the <strong>–</strong> symbol next to the trees TechnicalContact, ContentContact and Metadata to collapse them): [[File:abcdResponseMandatoryElements.png]] If you get an error, please check out the [[Debugging]] tutorial on how to debug a web service. You should continue the mapping process only after your web service returns results for the basal mapping we’ve just created. == Adding additional ABCD Elements == Once you’ve successfully managed to map the mandatory ABCD elements ("successfully" means the web service returns ABCD documents), you can go on with the mapping process. This is usually an iterative process: You add a handful of elements (for example, country, country code, locality text and coordinates for the gathering site), then you use the Query Form to see if the information ends up where it is supposed to do. This is the reason the Query Form opens up in a separate tab (or window, of you configured your web browser so): In one tab, you use the mapping editor to add new elements; after pressing <small>Save</small> you go to the Query Form tab and see what the response documents look like. Redo this until you’ve mapped all information you want to publish. In order to map new ABCD concepts, tick the check box labelled <small>Show all concepts</small> and press <small>Refresh</small>. The mapping editor will then display the whole ABCD tree, which looks admittedly a bit intimidating. In order to find the concept you’re looking for, use the search function of your browser. If you want to map the country name of the gathering site, for example, hit <small>Ctrl + F</small> (for Firefox, Microsoft IE and Opera) and type in the name of the ABCD element, namely ''country''. The browser will jump to the ''Country'' tree of the ''Gathering'' element, where you’ll find the elements for country name (''Name''), ISO country code (''ISO3166Code'') and the country name in the country’s language (''NameDerived''): [[File:abcdMappingEditorSearch.png|500px]] Continue the sequence mapping new elements/testing until you’ve mapped all information you want to publish. You can have a look at the list of recommend and commonly used ABCD elements on the [[CommonABCD2Concepts]] page. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki]. The following page will show you a sample response ABCD document for a web service with 77 concepts mapped. Of course your documents will look different, because you will probably use different ABCD elements, but still it’s good to get an idea. For easier viewing, you can save the contents of the box into a file with the extension .xml and open that in Firefix or MS Internet Explorer (don't use Safari). You will then be able to collapse the different XML sub trees: [[SampleABCDDocument]]. a19711e1118df2906251484584cc6ad04b8a1556 350 349 2011-08-18T11:50:04Z JoergHoletschek 3 /* Adding Mandatory ABCD Elements */ wikitext text/x-wiki This tutorial will explain how to set up a mapping for the ABCD2.06 data schema. If you intend to map another schema (e.g. one of the ABCD extension ABCD-EFG, ABCD-DNA or HISPID), you still should take the time to read it. Large portions of the process will be identical. == Access to Biological Collection Data Schema (ABCD) == ABCD in its current version 2.06 is the data schema typically used in conjunction with BioCASe. It is a highly complex XML data schema with about 1,000 elements that is able to store almost every piece of information that can be found in a natural history specimen collection or an observation database. The complexity is due to the fact that it can be used for a wide range of collections/databases – for living and preserved specimens, observations and culture collections, for zoological, botanical, bacterial and viral collections, marine or terrestrial, for herbaria, botanic and zoological gardens. For each of these special types, ABCD features special sections where information specific for this type can be stored. Other sections of ABCD will be shared by all these types, for example for gathering site, identifier or metadata. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a list of commonly used ABCD elements on the Wiki page [[CommonABCD2Concepts|Common ABCD2 Concepts]]. == Creating an ABCD Mapping for a Data Source == If you created a data source from a template (e.g. ''abcdmetadata''), a schema mapping for ABCD will be already existent. In this case, you can directly jump to the next section. If you created an empty data source, the section <small>Schemas</small> in the data source configuration overview will be empty: [[File:abcdSchemaMappings.png]] In order to create a new schema mapping, select the desired schema from the list and click <small>Create</small>. For ABCD, choose <small>ABCD_2.06.xml</small> (be careful not to hit one of the ABCD extensions, they look quite similar). After clicking <small>Create</small>, you will be directed to the mapping editor, which will allow you to edit the mappings. == The Mapping Editor == The mapping editor allows you to add, remove and edit ABCD mappings. On top you’ll find some summary information, on the bottom you’ll see the structure of the ABCD mapping: [[File:abcdMappingEditorEmpty.png]] First thing we will do is to set the root table, which is the table that holds the records you want to publish (specimens or observations). That table must hold one record per object to be published, identified by a unique identifier (called ''UnitID''). This identifier doesn't need to be the primary key necessarily, as long as it is unique (if that sentence confuses you, ignore it and just keep in mind that the UnitID must be unique.) Use the drop down box labelled <small>Root table Alias</small> to select the root table of your data model. If your metadata table holds just one record (because all occurrences published share the same metadata) and if it is not linked to the root table by a foreign key, you can specify this table in the drop-down box <small>Static table Alias</small>. The Provider Software will use a natural join to connect the metadata record stored in this table to all occurrences published. If your metadata table is referenced by a foreign key in the root able, you must not declare the metadata table as a static table! So either join the metadata table using a foreign key OR declare it as a static table. When you’ve set the root table (and static table, if you use that), press <small>Save</small>. As with the DB structure setup, this will write your changes to the configuration files, closing the tab without saving will discard your changes! Also, in case you messed something up, you can use the <small>Revert</small> button to restore the state when you’ve saved for the last time. In the mapping editor, the link <small>Overview</small> at the top of the page will take you back to the datasource configuration overview. After creating a mapping, it will appear in the section <small>Schemas</small>. The entry will show the number of mapped elements (0 for now) and the schema namespace. Pressing the trash can symbol will move it to the trash can, which means that your web service does not support this schema anymore; if you did this by mistake, you can restore it by pressing the <small>Restore</small> button. <small>Purge</small> will delete the schema mapping permanently: [[File:abcdSchemaMappings2.png]] You can map several schemas for a given datasource. For example, in addition to the current ABCD2 standard, you could add the deprecated version 1.2 (even though I have no idea why you should do that) or DarwinCore. Just select the appropriate schema in the drop-down list and click <small>Create</small>. Since the namespace of a schema will be used in requests, you cannot create two mappings for the same namespace. Trying this will get you an error message. == Adding Mandatory ABCD Elements == Clicking on a schema in the datasource overview page will take you again to the mapping editor. In the lower half of the page, you will see the basic structure of an ABCD document: [[File:abcdMappingEditorEmptyBottom.png|border]] <strong>Technical/Content Contact:</strong> These trees hold information on who should be contacted for technical issues or questions concerning the published database, each with name, address, email address and telephone number. <strong>Metadata:</strong> In this sub tree you’ll find all metadata fields, for example title and description of the dataset, ownership information, date last modified, IPR statements etc. <strong>Units:</strong> This part stores the "real" data you want to publish, that means all specimen or observation data. The three elements already shown in the tree are identifiers that make up the so called "Triple ID", which uniquely identifies records within primary biodiversity networks such as BioCASe and GBIF: * ''UnitID'', which is a unique identifier for each record to be published (you can use the catalogue number, for example, or an existing barcode); * ''SourceInstitutionID'', an identifier for the institution publishing the dataset (for example BGBM); * ''SourceID'', an identifier for the dataset (e.g. “Herbarium Berolinense’’). The elements already shown in the mapping editor are printed in red because they are mandatory, which means they are required for an ABCD document. Without them being mapped (to database fields that are non-empty), the Provider Software will not be able to construct valid ABCD documents and will not publish any records. So the first elements that should be mapped are the mandatory fields. To add a mapping, click the <strong>+</strong> symbol next to the field, which will open the mapping editor dialog, where you can select the database table/column you want to map the ABCD element to: [[File:abcdMappingEditorDialog.png]] In this case, the element ''/DataSets/DataSet/Metadata/RevisionData/DateModified'' is mapped to the column ''source_update'' of the ''metadata'' table, which has the data type ''date''. (Remember that table/column retrieval is not supported for all DBMS, so if you’re unlucky, you might need to type in table/column name manually.) <small>Save</small> will accept your settings and close the dialog, <small>Cancel</small> will discard any changes. Instead of mapping an element to a database field, you can map it to a literal. For example, you could enter the technical and content contact information directly as literals in the mapping editor dialog (assuming they’re the same for all records). However, we strongly discourage you from doing this. For one thing, this is just bad style, since it will scatter the information published over different places: Some will be retrieved from the database, some (the literals) will be loaded from the configuration files. Changing the metadata of your service later would then require access to the database and access to the BioCASe configuration – that means the configuration tool password. Be nice to the person who will do these changes and put everything neatly in a database table! The second reason is even more important: Using literals for an element will result in this element to become not searchable, which means it cannot be used in a filter of a Search or a Scan request. Even though this behaviour is discoverable with a Capabilities request (by evaluating the ''searchable'' flag), a potential user of your web service might just not think of that, so it’s a good idea to avoid that in first place. Please use literals sparsely! In the mapping editor window, you can select several source columns and several literals, even a combination of both. The Provider Software will simply concatenate the values retrieved from the database and the literals specified (in the order they’re given). But again: Keep in mind that this will make the element not searchable. A better solution is to do any concatenation operations in a database view. Use the mapping editor window to add mappings for all eight mandatory fields: Click on the <strong>+</strong> symbol next to the element, choose table/column name and the correct data type, then press <small>Save</small>. When you’re done, the page should look similar to the following image. Press <small>Save</small> in the main window to write your changes to the configuration files. [[File:abcdMappingEditorMandatoryElements.png]] == Initial Testing of the Data Source == Once you’ve mapped all mandatory elements, you can do the first test of your BioCASe web service. Even though you haven’t mapped any real information yet, it is a good idea to check the service and solve any problems hitherto before continuing with the mapping. Click on <small>Test mapping!</small> to open the Query Form that can be used for this purpose: [[File:abcdQueryFormEmpty.png]] With the Query Form you can send a BioCASe request to your web service; it will wait for the reply of the web service and display the response document. On top of the page you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ''flora'' being the name of the datasource. Make sure this is correct for your installation! Into the text box below you can enter the request XML text. Because it would be cumbersome to enter the whole BioCASe request, you can use the links below the box to load templates. For now, click on <small>ABCD2 Search</small> to fill the box with a BioCASe Search request for documents in the ABCD2 format (the one we’ve just mapped): <syntaxhighlight> <?xml version='1.0' encoding='UTF-8'?> <request xmlns='http://www.biocase.org/schemas/protocol/1.3'> <header><type>search</type></header> <search> <requestFormat>http://www.tdwg.org/schemas/abcd/2.06</requestFormat> <responseFormat start='0' limit='10'>http://www.tdwg.org/schemas/abcd/2.06</responseFormat> <filter> <like path='/DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString'>A*</like> </filter> <count>false</count> </search> </request> </syntaxhighlight> As you can see, there is a filter set on the ''ScientificName'' element. Since we haven’t mapped this element yet, this won’t work. So let’s remove the filter (all lines between <filter> and </filter>), leaving the text box like this: [[File:abcdQueryFormFilled.png]] Pressing <small>Submit</small> will send the request to the web service and – hopefully – display the response document with the correct ABCD records. For the mapping we’ve just created, it should look similar to the following document (use the <strong>–</strong> symbol next to the trees TechnicalContact, ContentContact and Metadata to collapse them): [[File:abcdResponseMandatoryElements.png]] If you get an error, please check out the [[Debugging]] tutorial on how to debug a web service. You should continue the mapping process only after your web service returns results for the basal mapping we’ve just created. == Adding additional ABCD Elements == Once you’ve successfully managed to map the mandatory ABCD elements ("successfully" means the web service returns ABCD documents), you can go on with the mapping process. This is usually an iterative process: You add a handful of elements (for example, country, country code, locality text and coordinates for the gathering site), then you use the Query Form to see if the information ends up where it is supposed to do. This is the reason the Query Form opens up in a separate tab (or window, of you configured your web browser so): In one tab, you use the mapping editor to add new elements; after pressing <small>Save</small> you go to the Query Form tab and see what the response documents look like. Redo this until you’ve mapped all information you want to publish. In order to map new ABCD concepts, tick the check box labelled <small>Show all concepts</small> and press <small>Refresh</small>. The mapping editor will then display the whole ABCD tree, which looks admittedly a bit intimidating. In order to find the concept you’re looking for, use the search function of your browser. If you want to map the country name of the gathering site, for example, hit <small>Ctrl + F</small> (for Firefox, Microsoft IE and Opera) and type in the name of the ABCD element, namely ''country''. The browser will jump to the ''Country'' tree of the ''Gathering'' element, where you’ll find the elements for country name (''Name''), ISO country code (''ISO3166Code'') and the country name in the country’s language (''NameDerived''): [[File:abcdMappingEditorSearch.png|500px]] Continue the sequence mapping new elements/testing until you’ve mapped all information you want to publish. You can have a look at the list of recommend and commonly used ABCD elements on the [[CommonABCD2Concepts]] page. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki]. The following page will show you a sample response ABCD document for a web service with 77 concepts mapped. Of course your documents will look different, because you will probably use different ABCD elements, but still it’s good to get an idea. For easier viewing, you can save the contents of the box into a file with the extension .xml and open that in Firefix or MS Internet Explorer (don't use Safari). You will then be able to collapse the different XML sub trees: [[SampleABCDDocument]]. 293b8cd9cd270ef1e9fdba81056d98c6525cd2a8 351 350 2011-08-18T11:53:35Z JoergHoletschek 3 /* Initial Testing of the Data Source */ wikitext text/x-wiki This tutorial will explain how to set up a mapping for the ABCD2.06 data schema. If you intend to map another schema (e.g. one of the ABCD extension ABCD-EFG, ABCD-DNA or HISPID), you still should take the time to read it. Large portions of the process will be identical. == Access to Biological Collection Data Schema (ABCD) == ABCD in its current version 2.06 is the data schema typically used in conjunction with BioCASe. It is a highly complex XML data schema with about 1,000 elements that is able to store almost every piece of information that can be found in a natural history specimen collection or an observation database. The complexity is due to the fact that it can be used for a wide range of collections/databases – for living and preserved specimens, observations and culture collections, for zoological, botanical, bacterial and viral collections, marine or terrestrial, for herbaria, botanic and zoological gardens. For each of these special types, ABCD features special sections where information specific for this type can be stored. Other sections of ABCD will be shared by all these types, for example for gathering site, identifier or metadata. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a list of commonly used ABCD elements on the Wiki page [[CommonABCD2Concepts|Common ABCD2 Concepts]]. == Creating an ABCD Mapping for a Data Source == If you created a data source from a template (e.g. ''abcdmetadata''), a schema mapping for ABCD will be already existent. In this case, you can directly jump to the next section. If you created an empty data source, the section <small>Schemas</small> in the data source configuration overview will be empty: [[File:abcdSchemaMappings.png]] In order to create a new schema mapping, select the desired schema from the list and click <small>Create</small>. For ABCD, choose <small>ABCD_2.06.xml</small> (be careful not to hit one of the ABCD extensions, they look quite similar). After clicking <small>Create</small>, you will be directed to the mapping editor, which will allow you to edit the mappings. == The Mapping Editor == The mapping editor allows you to add, remove and edit ABCD mappings. On top you’ll find some summary information, on the bottom you’ll see the structure of the ABCD mapping: [[File:abcdMappingEditorEmpty.png]] First thing we will do is to set the root table, which is the table that holds the records you want to publish (specimens or observations). That table must hold one record per object to be published, identified by a unique identifier (called ''UnitID''). This identifier doesn't need to be the primary key necessarily, as long as it is unique (if that sentence confuses you, ignore it and just keep in mind that the UnitID must be unique.) Use the drop down box labelled <small>Root table Alias</small> to select the root table of your data model. If your metadata table holds just one record (because all occurrences published share the same metadata) and if it is not linked to the root table by a foreign key, you can specify this table in the drop-down box <small>Static table Alias</small>. The Provider Software will use a natural join to connect the metadata record stored in this table to all occurrences published. If your metadata table is referenced by a foreign key in the root able, you must not declare the metadata table as a static table! So either join the metadata table using a foreign key OR declare it as a static table. When you’ve set the root table (and static table, if you use that), press <small>Save</small>. As with the DB structure setup, this will write your changes to the configuration files, closing the tab without saving will discard your changes! Also, in case you messed something up, you can use the <small>Revert</small> button to restore the state when you’ve saved for the last time. In the mapping editor, the link <small>Overview</small> at the top of the page will take you back to the datasource configuration overview. After creating a mapping, it will appear in the section <small>Schemas</small>. The entry will show the number of mapped elements (0 for now) and the schema namespace. Pressing the trash can symbol will move it to the trash can, which means that your web service does not support this schema anymore; if you did this by mistake, you can restore it by pressing the <small>Restore</small> button. <small>Purge</small> will delete the schema mapping permanently: [[File:abcdSchemaMappings2.png]] You can map several schemas for a given datasource. For example, in addition to the current ABCD2 standard, you could add the deprecated version 1.2 (even though I have no idea why you should do that) or DarwinCore. Just select the appropriate schema in the drop-down list and click <small>Create</small>. Since the namespace of a schema will be used in requests, you cannot create two mappings for the same namespace. Trying this will get you an error message. == Adding Mandatory ABCD Elements == Clicking on a schema in the datasource overview page will take you again to the mapping editor. In the lower half of the page, you will see the basic structure of an ABCD document: [[File:abcdMappingEditorEmptyBottom.png|border]] <strong>Technical/Content Contact:</strong> These trees hold information on who should be contacted for technical issues or questions concerning the published database, each with name, address, email address and telephone number. <strong>Metadata:</strong> In this sub tree you’ll find all metadata fields, for example title and description of the dataset, ownership information, date last modified, IPR statements etc. <strong>Units:</strong> This part stores the "real" data you want to publish, that means all specimen or observation data. The three elements already shown in the tree are identifiers that make up the so called "Triple ID", which uniquely identifies records within primary biodiversity networks such as BioCASe and GBIF: * ''UnitID'', which is a unique identifier for each record to be published (you can use the catalogue number, for example, or an existing barcode); * ''SourceInstitutionID'', an identifier for the institution publishing the dataset (for example BGBM); * ''SourceID'', an identifier for the dataset (e.g. “Herbarium Berolinense’’). The elements already shown in the mapping editor are printed in red because they are mandatory, which means they are required for an ABCD document. Without them being mapped (to database fields that are non-empty), the Provider Software will not be able to construct valid ABCD documents and will not publish any records. So the first elements that should be mapped are the mandatory fields. To add a mapping, click the <strong>+</strong> symbol next to the field, which will open the mapping editor dialog, where you can select the database table/column you want to map the ABCD element to: [[File:abcdMappingEditorDialog.png]] In this case, the element ''/DataSets/DataSet/Metadata/RevisionData/DateModified'' is mapped to the column ''source_update'' of the ''metadata'' table, which has the data type ''date''. (Remember that table/column retrieval is not supported for all DBMS, so if you’re unlucky, you might need to type in table/column name manually.) <small>Save</small> will accept your settings and close the dialog, <small>Cancel</small> will discard any changes. Instead of mapping an element to a database field, you can map it to a literal. For example, you could enter the technical and content contact information directly as literals in the mapping editor dialog (assuming they’re the same for all records). However, we strongly discourage you from doing this. For one thing, this is just bad style, since it will scatter the information published over different places: Some will be retrieved from the database, some (the literals) will be loaded from the configuration files. Changing the metadata of your service later would then require access to the database and access to the BioCASe configuration – that means the configuration tool password. Be nice to the person who will do these changes and put everything neatly in a database table! The second reason is even more important: Using literals for an element will result in this element to become not searchable, which means it cannot be used in a filter of a Search or a Scan request. Even though this behaviour is discoverable with a Capabilities request (by evaluating the ''searchable'' flag), a potential user of your web service might just not think of that, so it’s a good idea to avoid that in first place. Please use literals sparsely! In the mapping editor window, you can select several source columns and several literals, even a combination of both. The Provider Software will simply concatenate the values retrieved from the database and the literals specified (in the order they’re given). But again: Keep in mind that this will make the element not searchable. A better solution is to do any concatenation operations in a database view. Use the mapping editor window to add mappings for all eight mandatory fields: Click on the <strong>+</strong> symbol next to the element, choose table/column name and the correct data type, then press <small>Save</small>. When you’re done, the page should look similar to the following image. Press <small>Save</small> in the main window to write your changes to the configuration files. [[File:abcdMappingEditorMandatoryElements.png]] == Initial Testing of the Data Source == Once you’ve mapped all mandatory elements, you can do the first test of your BioCASe web service. Even though you haven’t mapped any real information yet, it is a good idea to check the service and solve any problems hitherto before continuing with the mapping. Click on <small>Test mapping!</small> to open the Query Form that can be used for this purpose: [[File:abcdQueryFormEmpty.png]] With the Query Form you can send a BioCASe request to your web service; it will wait for the reply of the web service and display the response document. On top of the page you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ''flora'' being the name of the datasource. Make sure this is correct for your installation! Into the text box below you can enter the request XML text. Because it would be cumbersome to enter the whole BioCASe request, you can use the links below the box to load templates. For now, click on <small>ABCD2 Search</small> to fill the box with a BioCASe Search request for documents in the ABCD2 format (the one you’ve just mapped): <syntaxhighlight> <?xml version='1.0' encoding='UTF-8'?> <request xmlns='http://www.biocase.org/schemas/protocol/1.3'> <header><type>search</type></header> <search> <requestFormat>http://www.tdwg.org/schemas/abcd/2.06</requestFormat> <responseFormat start='0' limit='10'>http://www.tdwg.org/schemas/abcd/2.06</responseFormat> <filter> <like path='/DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString'>A*</like> </filter> <count>false</count> </search> </request> </syntaxhighlight> As you can see, there is a filter set on the ''ScientificName'' element. Since we haven’t mapped this element yet, this won’t work. So let’s remove the filter (all lines between <filter> and </filter>), leaving the text box like this: [[File:abcdQueryFormFilled.png]] Pressing <small>Submit</small> will send the request to the web service and – hopefully – display the response document with the correct ABCD records. For the mapping we’ve just created, it should look similar to the following document (use the <strong>–</strong> symbol next to the trees TechnicalContact, ContentContact and Metadata to collapse them): [[File:abcdResponseMandatoryElements.png]] If you get an error, please check out the [[Debugging]] tutorial on how to debug a web service. You should continue the mapping process only after your web service returns results for the basal mapping we’ve just created. == Adding additional ABCD Elements == Once you’ve successfully managed to map the mandatory ABCD elements ("successfully" means the web service returns ABCD documents), you can go on with the mapping process. This is usually an iterative process: You add a handful of elements (for example, country, country code, locality text and coordinates for the gathering site), then you use the Query Form to see if the information ends up where it is supposed to do. This is the reason the Query Form opens up in a separate tab (or window, of you configured your web browser so): In one tab, you use the mapping editor to add new elements; after pressing <small>Save</small> you go to the Query Form tab and see what the response documents look like. Redo this until you’ve mapped all information you want to publish. In order to map new ABCD concepts, tick the check box labelled <small>Show all concepts</small> and press <small>Refresh</small>. The mapping editor will then display the whole ABCD tree, which looks admittedly a bit intimidating. In order to find the concept you’re looking for, use the search function of your browser. If you want to map the country name of the gathering site, for example, hit <small>Ctrl + F</small> (for Firefox, Microsoft IE and Opera) and type in the name of the ABCD element, namely ''country''. The browser will jump to the ''Country'' tree of the ''Gathering'' element, where you’ll find the elements for country name (''Name''), ISO country code (''ISO3166Code'') and the country name in the country’s language (''NameDerived''): [[File:abcdMappingEditorSearch.png|500px]] Continue the sequence mapping new elements/testing until you’ve mapped all information you want to publish. You can have a look at the list of recommend and commonly used ABCD elements on the [[CommonABCD2Concepts]] page. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki]. The following page will show you a sample response ABCD document for a web service with 77 concepts mapped. Of course your documents will look different, because you will probably use different ABCD elements, but still it’s good to get an idea. For easier viewing, you can save the contents of the box into a file with the extension .xml and open that in Firefix or MS Internet Explorer (don't use Safari). You will then be able to collapse the different XML sub trees: [[SampleABCDDocument]]. 68d3889706e3b461b699ccf08bc751a327715b4e 352 351 2011-08-18T11:55:23Z JoergHoletschek 3 /* Initial Testing of the Data Source */ wikitext text/x-wiki This tutorial will explain how to set up a mapping for the ABCD2.06 data schema. If you intend to map another schema (e.g. one of the ABCD extension ABCD-EFG, ABCD-DNA or HISPID), you still should take the time to read it. Large portions of the process will be identical. == Access to Biological Collection Data Schema (ABCD) == ABCD in its current version 2.06 is the data schema typically used in conjunction with BioCASe. It is a highly complex XML data schema with about 1,000 elements that is able to store almost every piece of information that can be found in a natural history specimen collection or an observation database. The complexity is due to the fact that it can be used for a wide range of collections/databases – for living and preserved specimens, observations and culture collections, for zoological, botanical, bacterial and viral collections, marine or terrestrial, for herbaria, botanic and zoological gardens. For each of these special types, ABCD features special sections where information specific for this type can be stored. Other sections of ABCD will be shared by all these types, for example for gathering site, identifier or metadata. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a list of commonly used ABCD elements on the Wiki page [[CommonABCD2Concepts|Common ABCD2 Concepts]]. == Creating an ABCD Mapping for a Data Source == If you created a data source from a template (e.g. ''abcdmetadata''), a schema mapping for ABCD will be already existent. In this case, you can directly jump to the next section. If you created an empty data source, the section <small>Schemas</small> in the data source configuration overview will be empty: [[File:abcdSchemaMappings.png]] In order to create a new schema mapping, select the desired schema from the list and click <small>Create</small>. For ABCD, choose <small>ABCD_2.06.xml</small> (be careful not to hit one of the ABCD extensions, they look quite similar). After clicking <small>Create</small>, you will be directed to the mapping editor, which will allow you to edit the mappings. == The Mapping Editor == The mapping editor allows you to add, remove and edit ABCD mappings. On top you’ll find some summary information, on the bottom you’ll see the structure of the ABCD mapping: [[File:abcdMappingEditorEmpty.png]] First thing we will do is to set the root table, which is the table that holds the records you want to publish (specimens or observations). That table must hold one record per object to be published, identified by a unique identifier (called ''UnitID''). This identifier doesn't need to be the primary key necessarily, as long as it is unique (if that sentence confuses you, ignore it and just keep in mind that the UnitID must be unique.) Use the drop down box labelled <small>Root table Alias</small> to select the root table of your data model. If your metadata table holds just one record (because all occurrences published share the same metadata) and if it is not linked to the root table by a foreign key, you can specify this table in the drop-down box <small>Static table Alias</small>. The Provider Software will use a natural join to connect the metadata record stored in this table to all occurrences published. If your metadata table is referenced by a foreign key in the root able, you must not declare the metadata table as a static table! So either join the metadata table using a foreign key OR declare it as a static table. When you’ve set the root table (and static table, if you use that), press <small>Save</small>. As with the DB structure setup, this will write your changes to the configuration files, closing the tab without saving will discard your changes! Also, in case you messed something up, you can use the <small>Revert</small> button to restore the state when you’ve saved for the last time. In the mapping editor, the link <small>Overview</small> at the top of the page will take you back to the datasource configuration overview. After creating a mapping, it will appear in the section <small>Schemas</small>. The entry will show the number of mapped elements (0 for now) and the schema namespace. Pressing the trash can symbol will move it to the trash can, which means that your web service does not support this schema anymore; if you did this by mistake, you can restore it by pressing the <small>Restore</small> button. <small>Purge</small> will delete the schema mapping permanently: [[File:abcdSchemaMappings2.png]] You can map several schemas for a given datasource. For example, in addition to the current ABCD2 standard, you could add the deprecated version 1.2 (even though I have no idea why you should do that) or DarwinCore. Just select the appropriate schema in the drop-down list and click <small>Create</small>. Since the namespace of a schema will be used in requests, you cannot create two mappings for the same namespace. Trying this will get you an error message. == Adding Mandatory ABCD Elements == Clicking on a schema in the datasource overview page will take you again to the mapping editor. In the lower half of the page, you will see the basic structure of an ABCD document: [[File:abcdMappingEditorEmptyBottom.png|border]] <strong>Technical/Content Contact:</strong> These trees hold information on who should be contacted for technical issues or questions concerning the published database, each with name, address, email address and telephone number. <strong>Metadata:</strong> In this sub tree you’ll find all metadata fields, for example title and description of the dataset, ownership information, date last modified, IPR statements etc. <strong>Units:</strong> This part stores the "real" data you want to publish, that means all specimen or observation data. The three elements already shown in the tree are identifiers that make up the so called "Triple ID", which uniquely identifies records within primary biodiversity networks such as BioCASe and GBIF: * ''UnitID'', which is a unique identifier for each record to be published (you can use the catalogue number, for example, or an existing barcode); * ''SourceInstitutionID'', an identifier for the institution publishing the dataset (for example BGBM); * ''SourceID'', an identifier for the dataset (e.g. “Herbarium Berolinense’’). The elements already shown in the mapping editor are printed in red because they are mandatory, which means they are required for an ABCD document. Without them being mapped (to database fields that are non-empty), the Provider Software will not be able to construct valid ABCD documents and will not publish any records. So the first elements that should be mapped are the mandatory fields. To add a mapping, click the <strong>+</strong> symbol next to the field, which will open the mapping editor dialog, where you can select the database table/column you want to map the ABCD element to: [[File:abcdMappingEditorDialog.png]] In this case, the element ''/DataSets/DataSet/Metadata/RevisionData/DateModified'' is mapped to the column ''source_update'' of the ''metadata'' table, which has the data type ''date''. (Remember that table/column retrieval is not supported for all DBMS, so if you’re unlucky, you might need to type in table/column name manually.) <small>Save</small> will accept your settings and close the dialog, <small>Cancel</small> will discard any changes. Instead of mapping an element to a database field, you can map it to a literal. For example, you could enter the technical and content contact information directly as literals in the mapping editor dialog (assuming they’re the same for all records). However, we strongly discourage you from doing this. For one thing, this is just bad style, since it will scatter the information published over different places: Some will be retrieved from the database, some (the literals) will be loaded from the configuration files. Changing the metadata of your service later would then require access to the database and access to the BioCASe configuration – that means the configuration tool password. Be nice to the person who will do these changes and put everything neatly in a database table! The second reason is even more important: Using literals for an element will result in this element to become not searchable, which means it cannot be used in a filter of a Search or a Scan request. Even though this behaviour is discoverable with a Capabilities request (by evaluating the ''searchable'' flag), a potential user of your web service might just not think of that, so it’s a good idea to avoid that in first place. Please use literals sparsely! In the mapping editor window, you can select several source columns and several literals, even a combination of both. The Provider Software will simply concatenate the values retrieved from the database and the literals specified (in the order they’re given). But again: Keep in mind that this will make the element not searchable. A better solution is to do any concatenation operations in a database view. Use the mapping editor window to add mappings for all eight mandatory fields: Click on the <strong>+</strong> symbol next to the element, choose table/column name and the correct data type, then press <small>Save</small>. When you’re done, the page should look similar to the following image. Press <small>Save</small> in the main window to write your changes to the configuration files. [[File:abcdMappingEditorMandatoryElements.png]] == Initial Testing of the Data Source == Once you’ve mapped all mandatory elements, you can do the first test of your BioCASe web service. Even though you haven’t mapped any real information yet, it is a good idea to check the service and solve any problems hitherto before continuing with the mapping. Click on <small>Test mapping!</small> to open the Query Form that can be used for this purpose: [[File:abcdQueryFormEmpty.png]] With the Query Form you can send a BioCASe request to your web service; it will wait for the reply of the web service and display the response document. On top of the page you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ''flora'' being the name of the datasource. Make sure this is correct for your installation! Into the text box below you can enter the request XML text. Because it would be cumbersome to enter the whole BioCASe request, you can use the links below the box to load templates. For now, click on <small>ABCD2 Search</small> to fill the box with a BioCASe Search request for documents in the ABCD2 format (the one you’ve just mapped): <syntaxhighlight> <?xml version='1.0' encoding='UTF-8'?> <request xmlns='http://www.biocase.org/schemas/protocol/1.3'> <header><type>search</type></header> <search> <requestFormat>http://www.tdwg.org/schemas/abcd/2.06</requestFormat> <responseFormat start='0' limit='10'>http://www.tdwg.org/schemas/abcd/2.06</responseFormat> <filter> <like path='/DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString'>A*</like> </filter> <count>false</count> </search> </request> </syntaxhighlight> As you can see, there is a filter set on the ''ScientificName'' element. Since we haven’t mapped this element yet, this won’t work. So let’s remove the filter (all lines between <filter> and </filter>, including the ''filter'' tags), leaving the text box like this: [[File:abcdQueryFormFilled.png]] Pressing <small>Submit</small> will send the request to the web service and – hopefully – display the response document with the correct ABCD records. For the mapping we’ve just created, it should look similar to the following document (use the <strong>–</strong> symbol next to the trees TechnicalContact, ContentContact and Metadata to collapse them): [[File:abcdResponseMandatoryElements.png]] If you get an error, please check out the [[Debugging]] tutorial on how to debug a web service. You should continue the mapping process only after your web service returns results for the basal mapping we’ve just created. == Adding additional ABCD Elements == Once you’ve successfully managed to map the mandatory ABCD elements ("successfully" means the web service returns ABCD documents), you can go on with the mapping process. This is usually an iterative process: You add a handful of elements (for example, country, country code, locality text and coordinates for the gathering site), then you use the Query Form to see if the information ends up where it is supposed to do. This is the reason the Query Form opens up in a separate tab (or window, of you configured your web browser so): In one tab, you use the mapping editor to add new elements; after pressing <small>Save</small> you go to the Query Form tab and see what the response documents look like. Redo this until you’ve mapped all information you want to publish. In order to map new ABCD concepts, tick the check box labelled <small>Show all concepts</small> and press <small>Refresh</small>. The mapping editor will then display the whole ABCD tree, which looks admittedly a bit intimidating. In order to find the concept you’re looking for, use the search function of your browser. If you want to map the country name of the gathering site, for example, hit <small>Ctrl + F</small> (for Firefox, Microsoft IE and Opera) and type in the name of the ABCD element, namely ''country''. The browser will jump to the ''Country'' tree of the ''Gathering'' element, where you’ll find the elements for country name (''Name''), ISO country code (''ISO3166Code'') and the country name in the country’s language (''NameDerived''): [[File:abcdMappingEditorSearch.png|500px]] Continue the sequence mapping new elements/testing until you’ve mapped all information you want to publish. You can have a look at the list of recommend and commonly used ABCD elements on the [[CommonABCD2Concepts]] page. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki]. The following page will show you a sample response ABCD document for a web service with 77 concepts mapped. Of course your documents will look different, because you will probably use different ABCD elements, but still it’s good to get an idea. For easier viewing, you can save the contents of the box into a file with the extension .xml and open that in Firefix or MS Internet Explorer (don't use Safari). You will then be able to collapse the different XML sub trees: [[SampleABCDDocument]]. 11b2ab52cc8fcda7fee05058f6111bbbf1ab74c5 353 352 2011-08-18T11:57:11Z JoergHoletschek 3 /* Initial Testing of the Data Source */ wikitext text/x-wiki This tutorial will explain how to set up a mapping for the ABCD2.06 data schema. If you intend to map another schema (e.g. one of the ABCD extension ABCD-EFG, ABCD-DNA or HISPID), you still should take the time to read it. Large portions of the process will be identical. == Access to Biological Collection Data Schema (ABCD) == ABCD in its current version 2.06 is the data schema typically used in conjunction with BioCASe. It is a highly complex XML data schema with about 1,000 elements that is able to store almost every piece of information that can be found in a natural history specimen collection or an observation database. The complexity is due to the fact that it can be used for a wide range of collections/databases – for living and preserved specimens, observations and culture collections, for zoological, botanical, bacterial and viral collections, marine or terrestrial, for herbaria, botanic and zoological gardens. For each of these special types, ABCD features special sections where information specific for this type can be stored. Other sections of ABCD will be shared by all these types, for example for gathering site, identifier or metadata. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a list of commonly used ABCD elements on the Wiki page [[CommonABCD2Concepts|Common ABCD2 Concepts]]. == Creating an ABCD Mapping for a Data Source == If you created a data source from a template (e.g. ''abcdmetadata''), a schema mapping for ABCD will be already existent. In this case, you can directly jump to the next section. If you created an empty data source, the section <small>Schemas</small> in the data source configuration overview will be empty: [[File:abcdSchemaMappings.png]] In order to create a new schema mapping, select the desired schema from the list and click <small>Create</small>. For ABCD, choose <small>ABCD_2.06.xml</small> (be careful not to hit one of the ABCD extensions, they look quite similar). After clicking <small>Create</small>, you will be directed to the mapping editor, which will allow you to edit the mappings. == The Mapping Editor == The mapping editor allows you to add, remove and edit ABCD mappings. On top you’ll find some summary information, on the bottom you’ll see the structure of the ABCD mapping: [[File:abcdMappingEditorEmpty.png]] First thing we will do is to set the root table, which is the table that holds the records you want to publish (specimens or observations). That table must hold one record per object to be published, identified by a unique identifier (called ''UnitID''). This identifier doesn't need to be the primary key necessarily, as long as it is unique (if that sentence confuses you, ignore it and just keep in mind that the UnitID must be unique.) Use the drop down box labelled <small>Root table Alias</small> to select the root table of your data model. If your metadata table holds just one record (because all occurrences published share the same metadata) and if it is not linked to the root table by a foreign key, you can specify this table in the drop-down box <small>Static table Alias</small>. The Provider Software will use a natural join to connect the metadata record stored in this table to all occurrences published. If your metadata table is referenced by a foreign key in the root able, you must not declare the metadata table as a static table! So either join the metadata table using a foreign key OR declare it as a static table. When you’ve set the root table (and static table, if you use that), press <small>Save</small>. As with the DB structure setup, this will write your changes to the configuration files, closing the tab without saving will discard your changes! Also, in case you messed something up, you can use the <small>Revert</small> button to restore the state when you’ve saved for the last time. In the mapping editor, the link <small>Overview</small> at the top of the page will take you back to the datasource configuration overview. After creating a mapping, it will appear in the section <small>Schemas</small>. The entry will show the number of mapped elements (0 for now) and the schema namespace. Pressing the trash can symbol will move it to the trash can, which means that your web service does not support this schema anymore; if you did this by mistake, you can restore it by pressing the <small>Restore</small> button. <small>Purge</small> will delete the schema mapping permanently: [[File:abcdSchemaMappings2.png]] You can map several schemas for a given datasource. For example, in addition to the current ABCD2 standard, you could add the deprecated version 1.2 (even though I have no idea why you should do that) or DarwinCore. Just select the appropriate schema in the drop-down list and click <small>Create</small>. Since the namespace of a schema will be used in requests, you cannot create two mappings for the same namespace. Trying this will get you an error message. == Adding Mandatory ABCD Elements == Clicking on a schema in the datasource overview page will take you again to the mapping editor. In the lower half of the page, you will see the basic structure of an ABCD document: [[File:abcdMappingEditorEmptyBottom.png|border]] <strong>Technical/Content Contact:</strong> These trees hold information on who should be contacted for technical issues or questions concerning the published database, each with name, address, email address and telephone number. <strong>Metadata:</strong> In this sub tree you’ll find all metadata fields, for example title and description of the dataset, ownership information, date last modified, IPR statements etc. <strong>Units:</strong> This part stores the "real" data you want to publish, that means all specimen or observation data. The three elements already shown in the tree are identifiers that make up the so called "Triple ID", which uniquely identifies records within primary biodiversity networks such as BioCASe and GBIF: * ''UnitID'', which is a unique identifier for each record to be published (you can use the catalogue number, for example, or an existing barcode); * ''SourceInstitutionID'', an identifier for the institution publishing the dataset (for example BGBM); * ''SourceID'', an identifier for the dataset (e.g. “Herbarium Berolinense’’). The elements already shown in the mapping editor are printed in red because they are mandatory, which means they are required for an ABCD document. Without them being mapped (to database fields that are non-empty), the Provider Software will not be able to construct valid ABCD documents and will not publish any records. So the first elements that should be mapped are the mandatory fields. To add a mapping, click the <strong>+</strong> symbol next to the field, which will open the mapping editor dialog, where you can select the database table/column you want to map the ABCD element to: [[File:abcdMappingEditorDialog.png]] In this case, the element ''/DataSets/DataSet/Metadata/RevisionData/DateModified'' is mapped to the column ''source_update'' of the ''metadata'' table, which has the data type ''date''. (Remember that table/column retrieval is not supported for all DBMS, so if you’re unlucky, you might need to type in table/column name manually.) <small>Save</small> will accept your settings and close the dialog, <small>Cancel</small> will discard any changes. Instead of mapping an element to a database field, you can map it to a literal. For example, you could enter the technical and content contact information directly as literals in the mapping editor dialog (assuming they’re the same for all records). However, we strongly discourage you from doing this. For one thing, this is just bad style, since it will scatter the information published over different places: Some will be retrieved from the database, some (the literals) will be loaded from the configuration files. Changing the metadata of your service later would then require access to the database and access to the BioCASe configuration – that means the configuration tool password. Be nice to the person who will do these changes and put everything neatly in a database table! The second reason is even more important: Using literals for an element will result in this element to become not searchable, which means it cannot be used in a filter of a Search or a Scan request. Even though this behaviour is discoverable with a Capabilities request (by evaluating the ''searchable'' flag), a potential user of your web service might just not think of that, so it’s a good idea to avoid that in first place. Please use literals sparsely! In the mapping editor window, you can select several source columns and several literals, even a combination of both. The Provider Software will simply concatenate the values retrieved from the database and the literals specified (in the order they’re given). But again: Keep in mind that this will make the element not searchable. A better solution is to do any concatenation operations in a database view. Use the mapping editor window to add mappings for all eight mandatory fields: Click on the <strong>+</strong> symbol next to the element, choose table/column name and the correct data type, then press <small>Save</small>. When you’re done, the page should look similar to the following image. Press <small>Save</small> in the main window to write your changes to the configuration files. [[File:abcdMappingEditorMandatoryElements.png]] == Initial Testing of the Data Source == Once you’ve mapped all mandatory elements, you can do the first test of your BioCASe web service. Even though you haven’t mapped any real information yet, it is a good idea to check the service and solve any problems hitherto before continuing with the mapping. Click on <small>Test mapping!</small> to open the Query Form that can be used for this purpose: [[File:abcdQueryFormEmpty.png]] With the Query Form you can send a BioCASe request to your web service; it will wait for the reply of the web service and display the response document. On top of the page you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ''flora'' being the name of the datasource. Make sure this is correct for your installation! Into the text box below you can enter the request XML text. Because it would be cumbersome to enter the whole BioCASe request, you can use the links below the box to load templates. For now, click on <small>ABCD2 Search</small> to fill the box with a BioCASe Search request for documents in the ABCD2 format (the one you’ve just mapped): <syntaxhighlight> <?xml version='1.0' encoding='UTF-8'?> <request xmlns='http://www.biocase.org/schemas/protocol/1.3'> <header><type>search</type></header> <search> <requestFormat>http://www.tdwg.org/schemas/abcd/2.06</requestFormat> <responseFormat start='0' limit='10'>http://www.tdwg.org/schemas/abcd/2.06</responseFormat> <filter> <like path='/DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString'>A*</like> </filter> <count>false</count> </search> </request> </syntaxhighlight> As you can see, there is a filter set on the ''ScientificName'' element. Since we haven’t mapped this element yet, this won’t work. So let’s remove the filter (all lines between <filter> and </filter>, including the ''filter'' tags), leaving the text box like this: [[File:abcdQueryFormFilled.png]] Pressing <small>Submit</small> will send the request to the web service and – hopefully – display the response document with the correct ABCD records. For the mapping we’ve just created, it should look similar to the following (use the "–" symbol next to the trees TechnicalContact, ContentContact and Metadata to collapse them): [[File:abcdResponseMandatoryElements.png]] If you get an error, please check out the [[Debugging]] tutorial on how to debug a web service. You should continue the mapping process only after your web service returns results for the basal mapping we’ve just created. == Adding additional ABCD Elements == Once you’ve successfully managed to map the mandatory ABCD elements ("successfully" means the web service returns ABCD documents), you can go on with the mapping process. This is usually an iterative process: You add a handful of elements (for example, country, country code, locality text and coordinates for the gathering site), then you use the Query Form to see if the information ends up where it is supposed to do. This is the reason the Query Form opens up in a separate tab (or window, of you configured your web browser so): In one tab, you use the mapping editor to add new elements; after pressing <small>Save</small> you go to the Query Form tab and see what the response documents look like. Redo this until you’ve mapped all information you want to publish. In order to map new ABCD concepts, tick the check box labelled <small>Show all concepts</small> and press <small>Refresh</small>. The mapping editor will then display the whole ABCD tree, which looks admittedly a bit intimidating. In order to find the concept you’re looking for, use the search function of your browser. If you want to map the country name of the gathering site, for example, hit <small>Ctrl + F</small> (for Firefox, Microsoft IE and Opera) and type in the name of the ABCD element, namely ''country''. The browser will jump to the ''Country'' tree of the ''Gathering'' element, where you’ll find the elements for country name (''Name''), ISO country code (''ISO3166Code'') and the country name in the country’s language (''NameDerived''): [[File:abcdMappingEditorSearch.png|500px]] Continue the sequence mapping new elements/testing until you’ve mapped all information you want to publish. You can have a look at the list of recommend and commonly used ABCD elements on the [[CommonABCD2Concepts]] page. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki]. The following page will show you a sample response ABCD document for a web service with 77 concepts mapped. Of course your documents will look different, because you will probably use different ABCD elements, but still it’s good to get an idea. For easier viewing, you can save the contents of the box into a file with the extension .xml and open that in Firefix or MS Internet Explorer (don't use Safari). You will then be able to collapse the different XML sub trees: [[SampleABCDDocument]]. d544ea1de53f01414f2821de591b14b5bc3704d2 354 353 2011-08-18T12:02:06Z JoergHoletschek 3 /* Adding additional ABCD Elements */ wikitext text/x-wiki This tutorial will explain how to set up a mapping for the ABCD2.06 data schema. If you intend to map another schema (e.g. one of the ABCD extension ABCD-EFG, ABCD-DNA or HISPID), you still should take the time to read it. Large portions of the process will be identical. == Access to Biological Collection Data Schema (ABCD) == ABCD in its current version 2.06 is the data schema typically used in conjunction with BioCASe. It is a highly complex XML data schema with about 1,000 elements that is able to store almost every piece of information that can be found in a natural history specimen collection or an observation database. The complexity is due to the fact that it can be used for a wide range of collections/databases – for living and preserved specimens, observations and culture collections, for zoological, botanical, bacterial and viral collections, marine or terrestrial, for herbaria, botanic and zoological gardens. For each of these special types, ABCD features special sections where information specific for this type can be stored. Other sections of ABCD will be shared by all these types, for example for gathering site, identifier or metadata. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a list of commonly used ABCD elements on the Wiki page [[CommonABCD2Concepts|Common ABCD2 Concepts]]. == Creating an ABCD Mapping for a Data Source == If you created a data source from a template (e.g. ''abcdmetadata''), a schema mapping for ABCD will be already existent. In this case, you can directly jump to the next section. If you created an empty data source, the section <small>Schemas</small> in the data source configuration overview will be empty: [[File:abcdSchemaMappings.png]] In order to create a new schema mapping, select the desired schema from the list and click <small>Create</small>. For ABCD, choose <small>ABCD_2.06.xml</small> (be careful not to hit one of the ABCD extensions, they look quite similar). After clicking <small>Create</small>, you will be directed to the mapping editor, which will allow you to edit the mappings. == The Mapping Editor == The mapping editor allows you to add, remove and edit ABCD mappings. On top you’ll find some summary information, on the bottom you’ll see the structure of the ABCD mapping: [[File:abcdMappingEditorEmpty.png]] First thing we will do is to set the root table, which is the table that holds the records you want to publish (specimens or observations). That table must hold one record per object to be published, identified by a unique identifier (called ''UnitID''). This identifier doesn't need to be the primary key necessarily, as long as it is unique (if that sentence confuses you, ignore it and just keep in mind that the UnitID must be unique.) Use the drop down box labelled <small>Root table Alias</small> to select the root table of your data model. If your metadata table holds just one record (because all occurrences published share the same metadata) and if it is not linked to the root table by a foreign key, you can specify this table in the drop-down box <small>Static table Alias</small>. The Provider Software will use a natural join to connect the metadata record stored in this table to all occurrences published. If your metadata table is referenced by a foreign key in the root able, you must not declare the metadata table as a static table! So either join the metadata table using a foreign key OR declare it as a static table. When you’ve set the root table (and static table, if you use that), press <small>Save</small>. As with the DB structure setup, this will write your changes to the configuration files, closing the tab without saving will discard your changes! Also, in case you messed something up, you can use the <small>Revert</small> button to restore the state when you’ve saved for the last time. In the mapping editor, the link <small>Overview</small> at the top of the page will take you back to the datasource configuration overview. After creating a mapping, it will appear in the section <small>Schemas</small>. The entry will show the number of mapped elements (0 for now) and the schema namespace. Pressing the trash can symbol will move it to the trash can, which means that your web service does not support this schema anymore; if you did this by mistake, you can restore it by pressing the <small>Restore</small> button. <small>Purge</small> will delete the schema mapping permanently: [[File:abcdSchemaMappings2.png]] You can map several schemas for a given datasource. For example, in addition to the current ABCD2 standard, you could add the deprecated version 1.2 (even though I have no idea why you should do that) or DarwinCore. Just select the appropriate schema in the drop-down list and click <small>Create</small>. Since the namespace of a schema will be used in requests, you cannot create two mappings for the same namespace. Trying this will get you an error message. == Adding Mandatory ABCD Elements == Clicking on a schema in the datasource overview page will take you again to the mapping editor. In the lower half of the page, you will see the basic structure of an ABCD document: [[File:abcdMappingEditorEmptyBottom.png|border]] <strong>Technical/Content Contact:</strong> These trees hold information on who should be contacted for technical issues or questions concerning the published database, each with name, address, email address and telephone number. <strong>Metadata:</strong> In this sub tree you’ll find all metadata fields, for example title and description of the dataset, ownership information, date last modified, IPR statements etc. <strong>Units:</strong> This part stores the "real" data you want to publish, that means all specimen or observation data. The three elements already shown in the tree are identifiers that make up the so called "Triple ID", which uniquely identifies records within primary biodiversity networks such as BioCASe and GBIF: * ''UnitID'', which is a unique identifier for each record to be published (you can use the catalogue number, for example, or an existing barcode); * ''SourceInstitutionID'', an identifier for the institution publishing the dataset (for example BGBM); * ''SourceID'', an identifier for the dataset (e.g. “Herbarium Berolinense’’). The elements already shown in the mapping editor are printed in red because they are mandatory, which means they are required for an ABCD document. Without them being mapped (to database fields that are non-empty), the Provider Software will not be able to construct valid ABCD documents and will not publish any records. So the first elements that should be mapped are the mandatory fields. To add a mapping, click the <strong>+</strong> symbol next to the field, which will open the mapping editor dialog, where you can select the database table/column you want to map the ABCD element to: [[File:abcdMappingEditorDialog.png]] In this case, the element ''/DataSets/DataSet/Metadata/RevisionData/DateModified'' is mapped to the column ''source_update'' of the ''metadata'' table, which has the data type ''date''. (Remember that table/column retrieval is not supported for all DBMS, so if you’re unlucky, you might need to type in table/column name manually.) <small>Save</small> will accept your settings and close the dialog, <small>Cancel</small> will discard any changes. Instead of mapping an element to a database field, you can map it to a literal. For example, you could enter the technical and content contact information directly as literals in the mapping editor dialog (assuming they’re the same for all records). However, we strongly discourage you from doing this. For one thing, this is just bad style, since it will scatter the information published over different places: Some will be retrieved from the database, some (the literals) will be loaded from the configuration files. Changing the metadata of your service later would then require access to the database and access to the BioCASe configuration – that means the configuration tool password. Be nice to the person who will do these changes and put everything neatly in a database table! The second reason is even more important: Using literals for an element will result in this element to become not searchable, which means it cannot be used in a filter of a Search or a Scan request. Even though this behaviour is discoverable with a Capabilities request (by evaluating the ''searchable'' flag), a potential user of your web service might just not think of that, so it’s a good idea to avoid that in first place. Please use literals sparsely! In the mapping editor window, you can select several source columns and several literals, even a combination of both. The Provider Software will simply concatenate the values retrieved from the database and the literals specified (in the order they’re given). But again: Keep in mind that this will make the element not searchable. A better solution is to do any concatenation operations in a database view. Use the mapping editor window to add mappings for all eight mandatory fields: Click on the <strong>+</strong> symbol next to the element, choose table/column name and the correct data type, then press <small>Save</small>. When you’re done, the page should look similar to the following image. Press <small>Save</small> in the main window to write your changes to the configuration files. [[File:abcdMappingEditorMandatoryElements.png]] == Initial Testing of the Data Source == Once you’ve mapped all mandatory elements, you can do the first test of your BioCASe web service. Even though you haven’t mapped any real information yet, it is a good idea to check the service and solve any problems hitherto before continuing with the mapping. Click on <small>Test mapping!</small> to open the Query Form that can be used for this purpose: [[File:abcdQueryFormEmpty.png]] With the Query Form you can send a BioCASe request to your web service; it will wait for the reply of the web service and display the response document. On top of the page you can see the URL of your web service. In this case it is http://localhost/biocase/pywrapper.cgi?dsa=flora, ''flora'' being the name of the datasource. Make sure this is correct for your installation! Into the text box below you can enter the request XML text. Because it would be cumbersome to enter the whole BioCASe request, you can use the links below the box to load templates. For now, click on <small>ABCD2 Search</small> to fill the box with a BioCASe Search request for documents in the ABCD2 format (the one you’ve just mapped): <syntaxhighlight> <?xml version='1.0' encoding='UTF-8'?> <request xmlns='http://www.biocase.org/schemas/protocol/1.3'> <header><type>search</type></header> <search> <requestFormat>http://www.tdwg.org/schemas/abcd/2.06</requestFormat> <responseFormat start='0' limit='10'>http://www.tdwg.org/schemas/abcd/2.06</responseFormat> <filter> <like path='/DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString'>A*</like> </filter> <count>false</count> </search> </request> </syntaxhighlight> As you can see, there is a filter set on the ''ScientificName'' element. Since we haven’t mapped this element yet, this won’t work. So let’s remove the filter (all lines between <filter> and </filter>, including the ''filter'' tags), leaving the text box like this: [[File:abcdQueryFormFilled.png]] Pressing <small>Submit</small> will send the request to the web service and – hopefully – display the response document with the correct ABCD records. For the mapping we’ve just created, it should look similar to the following (use the "–" symbol next to the trees TechnicalContact, ContentContact and Metadata to collapse them): [[File:abcdResponseMandatoryElements.png]] If you get an error, please check out the [[Debugging]] tutorial on how to debug a web service. You should continue the mapping process only after your web service returns results for the basal mapping we’ve just created. == Adding additional ABCD Elements == Once you’ve successfully managed to map the mandatory ABCD elements ("successfully" means the web service returns ABCD documents), you can go on with the mapping process. This is usually an iterative process: You add a handful of elements (for example, country, country code, locality text and coordinates for the gathering site), then you use the Query Form to see if the information ends up where it is supposed to do. This is the reason the Query Form opens up in a separate tab (or window, of you configured your web browser so): In one tab, you use the mapping editor to add new elements; after pressing <small>Save</small> you go to the Query Form tab and see what the response documents look like. Redo this until you’ve mapped all information you want to publish. In order to map new ABCD concepts, tick the check box labelled <small>Show all concepts</small> and press <small>Refresh</small>. The mapping editor will then display the whole ABCD tree, which looks admittedly a bit intimidating. In order to find the concept you’re looking for, use the search function of your browser. If you want to map the country name of the gathering site, for example, hit <small>Ctrl + F</small> (for Firefox, Microsoft IE and Opera) and type in the name of the ABCD element, namely ''country''. The browser will jump to the ''Country'' tree of the ''Gathering'' element, where you’ll find the elements for country name (''Name''), ISO country code (''ISO3166Code'') and the country name in the country’s language (''NameDerived''): [[File:abcdMappingEditorSearch.png|500px]] Continue the sequence mapping new elements/testing until you’ve mapped all information you want to publish. You can have a look at the list of recommend and commonly used ABCD elements on the [[CommonABCD2Concepts|Common ABCD2 Concepts]] page. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki]. The following page will show you a sample response ABCD document for a web service with 77 concepts mapped. Of course your documents will look different, because you will probably use different ABCD elements, but still it’s good to get an idea. For easier viewing, you can save the contents of the box into a file with the extension .xml and open that in Firefix or MS Internet Explorer (don't use Safari). You will then be able to collapse the different XML sub trees: [[SampleABCDDocument]]. 8069e0816e282c8899b887d5f2a072d3753aece2 355 354 2011-08-18T12:10:13Z JoergHoletschek 3 /* Initial Testing of the Data Source */ wikitext text/x-wiki This tutorial will explain how to set up a mapping for the ABCD2.06 data schema. If you intend to map another schema (e.g. one of the ABCD extension ABCD-EFG, ABCD-DNA or HISPID), you still should take the time to read it. Large portions of the process will be identical. == Access to Biological Collection Data Schema (ABCD) == ABCD in its current version 2.06 is the data schema typically used in conjunction with BioCASe. It is a highly complex XML data schema with about 1,000 elements that is able to store almost every piece of information that can be found in a natural history specimen collection or an observation database. The complexity is due to the fact that it can be used for a wide range of collections/databases – for living and preserved specimens, observations and culture collections, for zoological, botanical, bacterial and viral collections, marine or terrestrial, for herbaria, botanic and zoological gardens. For each of these special types, ABCD features special sections where information specific for this type can be stored. Other sections of ABCD will be shared by all these types, for example for gathering site, identifier or metadata. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a list of commonly used ABCD elements on the Wiki page [[CommonABCD2Concepts|Common ABCD2 Concepts]]. == Creating an ABCD Mapping for a Data Source == If you created a data source from a template (e.g. ''abcdmetadata''), a schema mapping for ABCD will be already existent. In this case, you can directly jump to the next section. If you created an empty data source, the section <small>Schemas</small> in the data source configuration overview will be empty: [[File:abcdSchemaMappings.png]] In order to create a new schema mapping, select the desired schema from the list and click <small>Create</small>. For ABCD, choose <small>ABCD_2.06.xml</small> (be careful not to hit one of the ABCD extensions, they look quite similar). After clicking <small>Create</small>, you will be directed to the mapping editor, which will allow you to edit the mappings. == The Mapping Editor == The mapping editor allows you to add, remove and edit ABCD mappings. On top you’ll find some summary information, on the bottom you’ll see the structure of the ABCD mapping: [[File:abcdMappingEditorEmpty.png]] First thing we will do is to set the root table, which is the table that holds the records you want to publish (specimens or observations). That table must hold one record per object to be published, identified by a unique identifier (called ''UnitID''). This identifier doesn't need to be the primary key necessarily, as long as it is unique (if that sentence confuses you, ignore it and just keep in mind that the UnitID must be unique.) Use the drop down box labelled <small>Root table Alias</small> to select the root table of your data model. If your metadata table holds just one record (because all occurrences published share the same metadata) and if it is not linked to the root table by a foreign key, you can specify this table in the drop-down box <small>Static table Alias</small>. The Provider Software will use a natural join to connect the metadata record stored in this table to all occurrences published. If your metadata table is referenced by a foreign key in the root able, you must not declare the metadata table as a static table! So either join the metadata table using a foreign key OR declare it as a static table. When you’ve set the root table (and static table, if you use that), press <small>Save</small>. As with the DB structure setup, this will write your changes to the configuration files, closing the tab without saving will discard your changes! Also, in case you messed something up, you can use the <small>Revert</small> button to restore the state when you’ve saved for the last time. In the mapping editor, the link <small>Overview</small> at the top of the page will take you back to the datasource configuration overview. After creating a mapping, it will appear in the section <small>Schemas</small>. The entry will show the number of mapped elements (0 for now) and the schema namespace. Pressing the trash can symbol will move it to the trash can, which means that your web service does not support this schema anymore; if you did this by mistake, you can restore it by pressing the <small>Restore</small> button. <small>Purge</small> will delete the schema mapping permanently: [[File:abcdSchemaMappings2.png]] You can map several schemas for a given datasource. For example, in addition to the current ABCD2 standard, you could add the deprecated version 1.2 (even though I have no idea why you should do that) or DarwinCore. Just select the appropriate schema in the drop-down list and click <small>Create</small>. Since the namespace of a schema will be used in requests, you cannot create two mappings for the same namespace. Trying this will get you an error message. == Adding Mandatory ABCD Elements == Clicking on a schema in the datasource overview page will take you again to the mapping editor. In the lower half of the page, you will see the basic structure of an ABCD document: [[File:abcdMappingEditorEmptyBottom.png|border]] <strong>Technical/Content Contact:</strong> These trees hold information on who should be contacted for technical issues or questions concerning the published database, each with name, address, email address and telephone number. <strong>Metadata:</strong> In this sub tree you’ll find all metadata fields, for example title and description of the dataset, ownership information, date last modified, IPR statements etc. <strong>Units:</strong> This part stores the "real" data you want to publish, that means all specimen or observation data. The three elements already shown in the tree are identifiers that make up the so called "Triple ID", which uniquely identifies records within primary biodiversity networks such as BioCASe and GBIF: * ''UnitID'', which is a unique identifier for each record to be published (you can use the catalogue number, for example, or an existing barcode); * ''SourceInstitutionID'', an identifier for the institution publishing the dataset (for example BGBM); * ''SourceID'', an identifier for the dataset (e.g. “Herbarium Berolinense’’). The elements already shown in the mapping editor are printed in red because they are mandatory, which means they are required for an ABCD document. Without them being mapped (to database fields that are non-empty), the Provider Software will not be able to construct valid ABCD documents and will not publish any records. So the first elements that should be mapped are the mandatory fields. To add a mapping, click the <strong>+</strong> symbol next to the field, which will open the mapping editor dialog, where you can select the database table/column you want to map the ABCD element to: [[File:abcdMappingEditorDialog.png]] In this case, the element ''/DataSets/DataSet/Metadata/RevisionData/DateModified'' is mapped to the column ''source_update'' of the ''metadata'' table, which has the data type ''date''. (Remember that table/column retrieval is not supported for all DBMS, so if you’re unlucky, you might need to type in table/column name manually.) <small>Save</small> will accept your settings and close the dialog, <small>Cancel</small> will discard any changes. Instead of mapping an element to a database field, you can map it to a literal. For example, you could enter the technical and content contact information directly as literals in the mapping editor dialog (assuming they’re the same for all records). However, we strongly discourage you from doing this. For one thing, this is just bad style, since it will scatter the information published over different places: Some will be retrieved from the database, some (the literals) will be loaded from the configuration files. Changing the metadata of your service later would then require access to the database and access to the BioCASe configuration – that means the configuration tool password. Be nice to the person who will do these changes and put everything neatly in a database table! The second reason is even more important: Using literals for an element will result in this element to become not searchable, which means it cannot be used in a filter of a Search or a Scan request. Even though this behaviour is discoverable with a Capabilities request (by evaluating the ''searchable'' flag), a potential user of your web service might just not think of that, so it’s a good idea to avoid that in first place. Please use literals sparsely! In the mapping editor window, you can select several source columns and several literals, even a combination of both. The Provider Software will simply concatenate the values retrieved from the database and the literals specified (in the order they’re given). But again: Keep in mind that this will make the element not searchable. A better solution is to do any concatenation operations in a database view. Use the mapping editor window to add mappings for all eight mandatory fields: Click on the <strong>+</strong> symbol next to the element, choose table/column name and the correct data type, then press <small>Save</small>. When you’re done, the page should look similar to the following image. Press <small>Save</small> in the main window to write your changes to the configuration files. [[File:abcdMappingEditorMandatoryElements.png]] == Initial Testing of the Data Source == Once you’ve mapped all mandatory elements, you can do the first test of your BioCASe web service. Even though you haven’t mapped any real information yet, it is a good idea to check the service and solve any problems hitherto before continuing with the mapping. Click on <small>Test mapping!</small> to open the Query Form that can be used for this purpose: [[File:abcdQueryFormEmpty.png]] With the Query Form you can send a BioCASe request to your web service; it will wait for the reply of the web service and display the response document. On top of the page you can see the URL of your web service. In this case it is ''<nowiki>http://localhost/biocase/pywrapper.cgi?dsa=flora</nowiki>'', ''flora'' being the name of the datasource. Make sure this is correct for your installation! Into the text box below you can enter the request XML text. Because it would be cumbersome to enter the whole BioCASe request, you can use the links below the box to load templates. For now, click on <small>ABCD2 Search</small> to fill the box with a BioCASe Search request for documents in the ABCD2 format (the one you’ve just mapped): <syntaxhighlight> <?xml version='1.0' encoding='UTF-8'?> <request xmlns='http://www.biocase.org/schemas/protocol/1.3'> <header><type>search</type></header> <search> <requestFormat>http://www.tdwg.org/schemas/abcd/2.06</requestFormat> <responseFormat start='0' limit='10'>http://www.tdwg.org/schemas/abcd/2.06</responseFormat> <filter> <like path='/DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString'>A*</like> </filter> <count>false</count> </search> </request> </syntaxhighlight> As you can see, there is a filter set on the ''ScientificName'' element. Since we haven’t mapped this element yet, this won’t work. So let’s remove the filter (all lines between <filter> and </filter>, including the ''filter'' tags), leaving the text box like this: [[File:abcdQueryFormFilled.png]] Pressing <small>Submit</small> will send the request to the web service and – hopefully – display the response document with the correct ABCD records. For the mapping we’ve just created, it should look similar to the following (use the "–" symbol next to the trees TechnicalContact, ContentContact and Metadata to collapse them): [[File:abcdResponseMandatoryElements.png]] If you get an error, please check out the [[Debugging]] tutorial on how to debug a web service. You should continue the mapping process only after your web service returns results for the basal mapping we’ve just created. == Adding additional ABCD Elements == Once you’ve successfully managed to map the mandatory ABCD elements ("successfully" means the web service returns ABCD documents), you can go on with the mapping process. This is usually an iterative process: You add a handful of elements (for example, country, country code, locality text and coordinates for the gathering site), then you use the Query Form to see if the information ends up where it is supposed to do. This is the reason the Query Form opens up in a separate tab (or window, of you configured your web browser so): In one tab, you use the mapping editor to add new elements; after pressing <small>Save</small> you go to the Query Form tab and see what the response documents look like. Redo this until you’ve mapped all information you want to publish. In order to map new ABCD concepts, tick the check box labelled <small>Show all concepts</small> and press <small>Refresh</small>. The mapping editor will then display the whole ABCD tree, which looks admittedly a bit intimidating. In order to find the concept you’re looking for, use the search function of your browser. If you want to map the country name of the gathering site, for example, hit <small>Ctrl + F</small> (for Firefox, Microsoft IE and Opera) and type in the name of the ABCD element, namely ''country''. The browser will jump to the ''Country'' tree of the ''Gathering'' element, where you’ll find the elements for country name (''Name''), ISO country code (''ISO3166Code'') and the country name in the country’s language (''NameDerived''): [[File:abcdMappingEditorSearch.png|500px]] Continue the sequence mapping new elements/testing until you’ve mapped all information you want to publish. You can have a look at the list of recommend and commonly used ABCD elements on the [[CommonABCD2Concepts|Common ABCD2 Concepts]] page. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki]. The following page will show you a sample response ABCD document for a web service with 77 concepts mapped. Of course your documents will look different, because you will probably use different ABCD elements, but still it’s good to get an idea. For easier viewing, you can save the contents of the box into a file with the extension .xml and open that in Firefix or MS Internet Explorer (don't use Safari). You will then be able to collapse the different XML sub trees: [[SampleABCDDocument]]. c7694dbb7613eed3a318e71330740ca8a2447a5f 0 41 356 283 2011-08-18T12:11:00Z JoergHoletschek 3 /* Basic Usage */ wikitext text/x-wiki This tutorial explains how to debug a BioCASe web service and a BioCASe installation in general. If you follow the instructions and don’t manage to get your service running, please [[Special:Contact|contact the BioCASe team]]. == Debugging a BioCASe Web Service == Debugging a BioCASe web service can be done with the manual Query Form. It allows you to load a BioCASe request (Search/Scan) from a template, edit the parameters of the request (e.g. the filter) and will show you the response document, which holds the requested ABCD records and useful debug information. === Opening the Query Form === There are several ways to open the Query Form. On the entry page of a datasource, <small>TEST and DEBUG</small> will open the Query Form: [[File:debugDatasourceEntryPage.png]] In the data source configuration tool, the link to the according query form can be always found in the link list at the top of the page. Moreover, in the mapping editor, the button <small>Test mapping!</small> will open the Query Form in a separate tab: [[File:abcdMappingEditorEmptyTop.png]] === Basic Usage === On top of the Query Form you can see the URL of your web service. In this case it is ''<nowiki>http://localhost/biocase/pywrapper.cgi?dsa=flora</nowiki>'', ''flora'' being the name of the datasource. Make sure this is the correct access point of the web service you want to debug! [[File:debugQueryformFull.png]] The usage of the Query Form is very straightforward: * Click on one of the links below the text box to load a template, for example for an ABCD2 Scan or ABCD2 Search; * Edit the request parameters; * Click on <small>Submit</small> to get the response document. The section <small>BioCASe protocol filter operators</small> lists the operators that can be used for creating complex filters that combine several filter criteria. More about creating filters can be found in the BioCASe Protocol documentation at http://www.biocase.org/products/protocols/index.shtml. Clicking on the <small>Show</small> link in the <small>Capabilities</small> section will show the Capabilities response in a frame of the Query Form. This is useful, since it shows the namespaces for the schemas supported by the web service (which is required in the request text) and XPaths of mapped ABCD elements (required for creating filters on these elements). By using Copy & Paste you can spare the trouble of entering these monstrous strings of characters. === Changing the Debug Level for a Data Source === Debug info can be set to one of four levels for a BioCASe data source: * <strong>Error</strong> is the highest level will only output errors that prevent the web server from answering a request. * <strong>Warning</strong> will output errors that keep BioCASe from answering a request and warnings that don’t prevent the fulfilling of the request, but should be analysed carefully. For example, they might tell you that records have been dropped because mandatory ABCD elements have not been mapped or were empty for certain records. * <strong>Info</strong> is the default and will add some more information about how the Provider Software processes the request. * <strong>Debug</strong> is the most detailed level and will produce very verbose debug output. It adds very comprehensive information about how the ABCD documents are constructed and is suitable only for experienced users. For most cases, keeping the default ''Info'' level should be sufficient for testing and debugging a web service. For production use, you should set it to ''Error'' later. You can do this in the configuration of a datasource under Settings: [[File:debugDatasourceSettings.png]] == Analyzing the Response Document == Upon clicking <small>Submit</small>, the Query Form will send the request to the web service and wait for the response. It might take some time – up to minutes – until the response document shows up. Amongst others, this depends on * the complexity of the query (especially the filters), * the size of the dataset returned, * the debug level set (“debug” results in very exhaustive debug output), * the performance of the servers running BioCASe and the database, * the network bandwidth. The response document consists of three sections (a full response document can be found [[CommonABCD2Concepts|here]]): [[File:debugResponseDocumentOverview.png]] === Header === This section contains information about the BioCASe installation – operating system, DBMS used, Python/BioCASe version etc. It is intended for a consumer application and won’t be very helpful for debugging: <syntaxhighlight> <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> </syntaxhighlight> === Content === This section makes up the payload of the response document, depending on the type of request: <strong>Capabilities:</strong> Lists all schemas supported by the web service. For each schema, all mapped concepts (identified by their XPaths) are listed with data type and searchable flag. <strong>Scan:</strong> The document will simply list all distinct values for the concept specified in the request. When done for the scientific name, it would look similar to this: <syntaxhighlight> <biocase:content recordCount="1664" recordDropped="0" recordStart="0" totalSearchHits="1664"> <biocase:scan> <biocase:value>Achillea atrata L.</biocase:value> <biocase:value>Achillea millefolium L.</biocase:value> <biocase:value>Achillea nobilis L. ssp. nobilis</biocase:value> <biocase:value>Achillea ptarmica L.</biocase:value> <biocase:value>Acinos alpinus (L.) Moench</biocase:value> <biocase:value>Acinos arvensis (Lam.) Dandy</biocase:value> <biocase:value>Aconitum degenii ssp. paniculatum (Arcang.) Mucher</biocase:value> <biocase:value>Aconitum lycoctonum L. ssp. lycoctonum</biocase:value> <biocase:value>Aconitum napellus L. s.l.</biocase:value> ... </biocase:scan> </biocase:content> </syntaxhighlight> <strong>Search:</strong> For this type of request, the content consists of the ABCD dataset(s). See the the [[CommonABCD2Concepts| ABCD2 example]]. === Diagnostics (Debug Output) === This section is the most interesting part for debugging. If debug level is set to ''Info'' (which is recommended for debugging), it will document the processing of the request by the web service. The output is easy to understand, for example it will contain the SQL statements sent to the database. For a successful search request it could look similar to this: <syntaxhighlight> <biocase:diagnostics> <biocase:diagnostic severity="INFO"> Datasource wrapper FloraExsiccataBavarica requested </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Reading PSF from C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica \provider_setup_file.xml </biocase:diagnostic> <biocase:diagnostic severity="DEBUG"> PSF: PSF=C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\provider_setup_file.xml, recLimit=100, loglevel=20, user=webuser, database=rbg, dbIP=192.168.2.10, dbms=mysql, encoding=latin_1, schemas={u'http://www.tdwg.org/schemas/abcd/2.06': <biocase.wrapper.psf_handler.SupportedSchema instance at 0x011F5B98>}, tablegraph=GRAPH: graph: unit-metadata, +++ ALIAS2TABLE: {u'unit': u'feb3', u'metadata': u'metadata'} </biocase:diagnostic> <biocase:diagnostic severity="INFO"> BioCASe protocol used. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> HTTP parameter 'query' used for building the request. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT DISTINCT unit.id FROM feb3 AS unit WHERE (unit.nameautor LIKE 'Allium %' ) LIMIT 11' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT unit.id, metadata.supplier_organisation, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (40, 41, 42, 43, 44, 45, 46, 47)) ORDER BY unit.id' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> time to execute request is 0.46599984169 </biocase:diagnostic> </biocase:diagnostics> </syntaxhighlight> == Sample Errors and Warnings in the Debug Output == <syntaxhighlight> <biocase:diagnostic severity="ERROR"> No DB connection could be established. Please verify DB state and connection parameters in your provider_setup_file.xml. </biocase:diagnostic> </syntaxhighlight> Well... just what it says. The Provider Software could not connect to the database to be published. Check the database connection settings (server IP, database name, user, and password) and make sure you’ve installed the Python package for the DBMS you’re using (see the <small>Libs test</small> page in <small>Utilities</small>). <syntaxhighlight> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\flora\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="ERROR"> The CMF DB mapping file http://www.tdwg.org/schemas/abcd/2.06 was corrupt or could not be interpreted. </biocase:diagnostic> </syntaxhighlight> The requested schema (identified by the namespace http://www.tdwg.org/schemas/abcd/2.06) is not supported by that service. Create a mapping and map elements. <syntaxhighlight> <biocase:diagnostic severity="ERROR"> The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result /TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts </biocase:diagnostic> </syntaxhighlight> Obvious - happens if you try to use a concept that is not mapped in a scan request or a filter. <syntaxhighlight> <biocase:diagnostic severity="ERROR"> A SQL statement produced an error: SELECT unit.id, metadata.supplier_organisation, metadata.supplier_email, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) ORDER BY unit.id </biocase:diagnostic> </syntaxhighlight> That could have a number of reasons: One of the table/column names that were set up in the configuration does not exist (because it was renamed or removed) or the credentials used by the BPS do not have sufficient privileges. Simply copy the SQL statement and execute it manually on the database with a regular database client that will show you the detailed error message returned by the DBMS. <syntaxhighlight> <biocase:diagnostic severity="WARNING"> The mandatory child element SourceInstitutionID could not be created [0 existing, but 1 required]. The element /DataSets/DataSet/Units/Unit was also dropped. </biocase:diagnostic> <biocase:diagnostic severity="WARNING"> The mandatory child element Units could not be created [0 existing, but 1 required]. The element /DataSets/DataSet was also dropped. </biocase:diagnostic> </syntaxhighlight> One problem caused the next one in this sequence of warnings: The mandatory ABCD element ''SourceInstitutionID '' was either not mapped or empty; therefore the respective record (''unit'') was dropped. Because this happened for all units in the dataset, all units were dropped, resulting in an empty ''Units'' sub tree. The second warning tells you what happened then: Because the ''Units'' element is mandatory for the ''Dataset'' element, the whole dataset was dropped. As a result, the response ABCD document was empty. == Debugging a BioCASe Installation == In case the debug output of the web service doesn’t help you to find the problem, or if the BPS behaves not as intended without displaying any error messages, you should have a look at the debug logs written by the BPS. To turn on the Debug logs, go to the System Administration page and set <small>Debug logs</small> to <small>True</small>. [[File:instConfigTool.png|500px]] Redo the step that didn’t work and open the ''log'' folder of your BioCASe installation. There are several global log files (pywrapper_request.log, pywrapper_error.log, webapp_debug.log, webapp_error.log) and a debug files per data source (e.g. debug_flora.log). Open these files and see if they can give you a clue to the problem. If you still can’t get it running, [[Special:Contact|contact the BioCASe team]]. a91df8e26166d9ed2107752f6c64e086c7de31db 357 356 2011-08-18T12:13:11Z JoergHoletschek 3 /* Analyzing the Response Document */ wikitext text/x-wiki This tutorial explains how to debug a BioCASe web service and a BioCASe installation in general. If you follow the instructions and don’t manage to get your service running, please [[Special:Contact|contact the BioCASe team]]. == Debugging a BioCASe Web Service == Debugging a BioCASe web service can be done with the manual Query Form. It allows you to load a BioCASe request (Search/Scan) from a template, edit the parameters of the request (e.g. the filter) and will show you the response document, which holds the requested ABCD records and useful debug information. === Opening the Query Form === There are several ways to open the Query Form. On the entry page of a datasource, <small>TEST and DEBUG</small> will open the Query Form: [[File:debugDatasourceEntryPage.png]] In the data source configuration tool, the link to the according query form can be always found in the link list at the top of the page. Moreover, in the mapping editor, the button <small>Test mapping!</small> will open the Query Form in a separate tab: [[File:abcdMappingEditorEmptyTop.png]] === Basic Usage === On top of the Query Form you can see the URL of your web service. In this case it is ''<nowiki>http://localhost/biocase/pywrapper.cgi?dsa=flora</nowiki>'', ''flora'' being the name of the datasource. Make sure this is the correct access point of the web service you want to debug! [[File:debugQueryformFull.png]] The usage of the Query Form is very straightforward: * Click on one of the links below the text box to load a template, for example for an ABCD2 Scan or ABCD2 Search; * Edit the request parameters; * Click on <small>Submit</small> to get the response document. The section <small>BioCASe protocol filter operators</small> lists the operators that can be used for creating complex filters that combine several filter criteria. More about creating filters can be found in the BioCASe Protocol documentation at http://www.biocase.org/products/protocols/index.shtml. Clicking on the <small>Show</small> link in the <small>Capabilities</small> section will show the Capabilities response in a frame of the Query Form. This is useful, since it shows the namespaces for the schemas supported by the web service (which is required in the request text) and XPaths of mapped ABCD elements (required for creating filters on these elements). By using Copy & Paste you can spare the trouble of entering these monstrous strings of characters. === Changing the Debug Level for a Data Source === Debug info can be set to one of four levels for a BioCASe data source: * <strong>Error</strong> is the highest level will only output errors that prevent the web server from answering a request. * <strong>Warning</strong> will output errors that keep BioCASe from answering a request and warnings that don’t prevent the fulfilling of the request, but should be analysed carefully. For example, they might tell you that records have been dropped because mandatory ABCD elements have not been mapped or were empty for certain records. * <strong>Info</strong> is the default and will add some more information about how the Provider Software processes the request. * <strong>Debug</strong> is the most detailed level and will produce very verbose debug output. It adds very comprehensive information about how the ABCD documents are constructed and is suitable only for experienced users. For most cases, keeping the default ''Info'' level should be sufficient for testing and debugging a web service. For production use, you should set it to ''Error'' later. You can do this in the configuration of a datasource under Settings: [[File:debugDatasourceSettings.png]] == Analyzing the Response Document == Upon clicking <small>Submit</small>, the Query Form will send the request to the web service and wait for the response. It might take some time – up to minutes – until the response document shows up. Amongst others, this depends on * the complexity of the query (especially the filters), * the size of the dataset returned, * the debug level set (“debug” results in very exhaustive debug output), * the performance of the servers running BioCASe and the database, * the network bandwidth. The response document consists of three sections (a full response document can be found [[SampleABCDDocument|here]]): [[File:debugResponseDocumentOverview.png]] === Header === This section contains information about the BioCASe installation – operating system, DBMS used, Python/BioCASe version etc. It is intended for a consumer application and won’t be very helpful for debugging: <syntaxhighlight> <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> </syntaxhighlight> === Content === This section makes up the payload of the response document, depending on the type of request: <strong>Capabilities:</strong> Lists all schemas supported by the web service. For each schema, all mapped concepts (identified by their XPaths) are listed with data type and searchable flag. <strong>Scan:</strong> The document will simply list all distinct values for the concept specified in the request. When done for the scientific name, it would look similar to this: <syntaxhighlight> <biocase:content recordCount="1664" recordDropped="0" recordStart="0" totalSearchHits="1664"> <biocase:scan> <biocase:value>Achillea atrata L.</biocase:value> <biocase:value>Achillea millefolium L.</biocase:value> <biocase:value>Achillea nobilis L. ssp. nobilis</biocase:value> <biocase:value>Achillea ptarmica L.</biocase:value> <biocase:value>Acinos alpinus (L.) Moench</biocase:value> <biocase:value>Acinos arvensis (Lam.) Dandy</biocase:value> <biocase:value>Aconitum degenii ssp. paniculatum (Arcang.) Mucher</biocase:value> <biocase:value>Aconitum lycoctonum L. ssp. lycoctonum</biocase:value> <biocase:value>Aconitum napellus L. s.l.</biocase:value> ... </biocase:scan> </biocase:content> </syntaxhighlight> <strong>Search:</strong> For this type of request, the content consists of the ABCD dataset(s). See the the [[CommonABCD2Concepts| ABCD2 example]]. === Diagnostics (Debug Output) === This section is the most interesting part for debugging. If debug level is set to ''Info'' (which is recommended for debugging), it will document the processing of the request by the web service. The output is easy to understand, for example it will contain the SQL statements sent to the database. For a successful search request it could look similar to this: <syntaxhighlight> <biocase:diagnostics> <biocase:diagnostic severity="INFO"> Datasource wrapper FloraExsiccataBavarica requested </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Reading PSF from C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica \provider_setup_file.xml </biocase:diagnostic> <biocase:diagnostic severity="DEBUG"> PSF: PSF=C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\provider_setup_file.xml, recLimit=100, loglevel=20, user=webuser, database=rbg, dbIP=192.168.2.10, dbms=mysql, encoding=latin_1, schemas={u'http://www.tdwg.org/schemas/abcd/2.06': <biocase.wrapper.psf_handler.SupportedSchema instance at 0x011F5B98>}, tablegraph=GRAPH: graph: unit-metadata, +++ ALIAS2TABLE: {u'unit': u'feb3', u'metadata': u'metadata'} </biocase:diagnostic> <biocase:diagnostic severity="INFO"> BioCASe protocol used. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> HTTP parameter 'query' used for building the request. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT DISTINCT unit.id FROM feb3 AS unit WHERE (unit.nameautor LIKE 'Allium %' ) LIMIT 11' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT unit.id, metadata.supplier_organisation, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (40, 41, 42, 43, 44, 45, 46, 47)) ORDER BY unit.id' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> time to execute request is 0.46599984169 </biocase:diagnostic> </biocase:diagnostics> </syntaxhighlight> == Sample Errors and Warnings in the Debug Output == <syntaxhighlight> <biocase:diagnostic severity="ERROR"> No DB connection could be established. Please verify DB state and connection parameters in your provider_setup_file.xml. </biocase:diagnostic> </syntaxhighlight> Well... just what it says. The Provider Software could not connect to the database to be published. Check the database connection settings (server IP, database name, user, and password) and make sure you’ve installed the Python package for the DBMS you’re using (see the <small>Libs test</small> page in <small>Utilities</small>). <syntaxhighlight> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\flora\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="ERROR"> The CMF DB mapping file http://www.tdwg.org/schemas/abcd/2.06 was corrupt or could not be interpreted. </biocase:diagnostic> </syntaxhighlight> The requested schema (identified by the namespace http://www.tdwg.org/schemas/abcd/2.06) is not supported by that service. Create a mapping and map elements. <syntaxhighlight> <biocase:diagnostic severity="ERROR"> The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result /TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts </biocase:diagnostic> </syntaxhighlight> Obvious - happens if you try to use a concept that is not mapped in a scan request or a filter. <syntaxhighlight> <biocase:diagnostic severity="ERROR"> A SQL statement produced an error: SELECT unit.id, metadata.supplier_organisation, metadata.supplier_email, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) ORDER BY unit.id </biocase:diagnostic> </syntaxhighlight> That could have a number of reasons: One of the table/column names that were set up in the configuration does not exist (because it was renamed or removed) or the credentials used by the BPS do not have sufficient privileges. Simply copy the SQL statement and execute it manually on the database with a regular database client that will show you the detailed error message returned by the DBMS. <syntaxhighlight> <biocase:diagnostic severity="WARNING"> The mandatory child element SourceInstitutionID could not be created [0 existing, but 1 required]. The element /DataSets/DataSet/Units/Unit was also dropped. </biocase:diagnostic> <biocase:diagnostic severity="WARNING"> The mandatory child element Units could not be created [0 existing, but 1 required]. The element /DataSets/DataSet was also dropped. </biocase:diagnostic> </syntaxhighlight> One problem caused the next one in this sequence of warnings: The mandatory ABCD element ''SourceInstitutionID '' was either not mapped or empty; therefore the respective record (''unit'') was dropped. Because this happened for all units in the dataset, all units were dropped, resulting in an empty ''Units'' sub tree. The second warning tells you what happened then: Because the ''Units'' element is mandatory for the ''Dataset'' element, the whole dataset was dropped. As a result, the response ABCD document was empty. == Debugging a BioCASe Installation == In case the debug output of the web service doesn’t help you to find the problem, or if the BPS behaves not as intended without displaying any error messages, you should have a look at the debug logs written by the BPS. To turn on the Debug logs, go to the System Administration page and set <small>Debug logs</small> to <small>True</small>. [[File:instConfigTool.png|500px]] Redo the step that didn’t work and open the ''log'' folder of your BioCASe installation. There are several global log files (pywrapper_request.log, pywrapper_error.log, webapp_debug.log, webapp_error.log) and a debug files per data source (e.g. debug_flora.log). Open these files and see if they can give you a clue to the problem. If you still can’t get it running, [[Special:Contact|contact the BioCASe team]]. c0f46e142cc9c4fd42069ce87768cf8c62085ad5 358 357 2011-08-18T12:15:01Z JoergHoletschek 3 /* Content */ wikitext text/x-wiki This tutorial explains how to debug a BioCASe web service and a BioCASe installation in general. If you follow the instructions and don’t manage to get your service running, please [[Special:Contact|contact the BioCASe team]]. == Debugging a BioCASe Web Service == Debugging a BioCASe web service can be done with the manual Query Form. It allows you to load a BioCASe request (Search/Scan) from a template, edit the parameters of the request (e.g. the filter) and will show you the response document, which holds the requested ABCD records and useful debug information. === Opening the Query Form === There are several ways to open the Query Form. On the entry page of a datasource, <small>TEST and DEBUG</small> will open the Query Form: [[File:debugDatasourceEntryPage.png]] In the data source configuration tool, the link to the according query form can be always found in the link list at the top of the page. Moreover, in the mapping editor, the button <small>Test mapping!</small> will open the Query Form in a separate tab: [[File:abcdMappingEditorEmptyTop.png]] === Basic Usage === On top of the Query Form you can see the URL of your web service. In this case it is ''<nowiki>http://localhost/biocase/pywrapper.cgi?dsa=flora</nowiki>'', ''flora'' being the name of the datasource. Make sure this is the correct access point of the web service you want to debug! [[File:debugQueryformFull.png]] The usage of the Query Form is very straightforward: * Click on one of the links below the text box to load a template, for example for an ABCD2 Scan or ABCD2 Search; * Edit the request parameters; * Click on <small>Submit</small> to get the response document. The section <small>BioCASe protocol filter operators</small> lists the operators that can be used for creating complex filters that combine several filter criteria. More about creating filters can be found in the BioCASe Protocol documentation at http://www.biocase.org/products/protocols/index.shtml. Clicking on the <small>Show</small> link in the <small>Capabilities</small> section will show the Capabilities response in a frame of the Query Form. This is useful, since it shows the namespaces for the schemas supported by the web service (which is required in the request text) and XPaths of mapped ABCD elements (required for creating filters on these elements). By using Copy & Paste you can spare the trouble of entering these monstrous strings of characters. === Changing the Debug Level for a Data Source === Debug info can be set to one of four levels for a BioCASe data source: * <strong>Error</strong> is the highest level will only output errors that prevent the web server from answering a request. * <strong>Warning</strong> will output errors that keep BioCASe from answering a request and warnings that don’t prevent the fulfilling of the request, but should be analysed carefully. For example, they might tell you that records have been dropped because mandatory ABCD elements have not been mapped or were empty for certain records. * <strong>Info</strong> is the default and will add some more information about how the Provider Software processes the request. * <strong>Debug</strong> is the most detailed level and will produce very verbose debug output. It adds very comprehensive information about how the ABCD documents are constructed and is suitable only for experienced users. For most cases, keeping the default ''Info'' level should be sufficient for testing and debugging a web service. For production use, you should set it to ''Error'' later. You can do this in the configuration of a datasource under Settings: [[File:debugDatasourceSettings.png]] == Analyzing the Response Document == Upon clicking <small>Submit</small>, the Query Form will send the request to the web service and wait for the response. It might take some time – up to minutes – until the response document shows up. Amongst others, this depends on * the complexity of the query (especially the filters), * the size of the dataset returned, * the debug level set (“debug” results in very exhaustive debug output), * the performance of the servers running BioCASe and the database, * the network bandwidth. The response document consists of three sections (a full response document can be found [[SampleABCDDocument|here]]): [[File:debugResponseDocumentOverview.png]] === Header === This section contains information about the BioCASe installation – operating system, DBMS used, Python/BioCASe version etc. It is intended for a consumer application and won’t be very helpful for debugging: <syntaxhighlight> <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> </syntaxhighlight> === Content === This section makes up the payload of the response document, depending on the type of request: <strong>Capabilities:</strong> Lists all schemas supported by the web service. For each schema, all mapped concepts (identified by their XPaths) are listed with data type and searchable flag. <strong>Scan:</strong> The document will simply list all distinct values for the concept specified in the request. When done for the scientific name, it would look similar to this: <syntaxhighlight> <biocase:content recordCount="1664" recordDropped="0" recordStart="0" totalSearchHits="1664"> <biocase:scan> <biocase:value>Achillea atrata L.</biocase:value> <biocase:value>Achillea millefolium L.</biocase:value> <biocase:value>Achillea nobilis L. ssp. nobilis</biocase:value> <biocase:value>Achillea ptarmica L.</biocase:value> <biocase:value>Acinos alpinus (L.) Moench</biocase:value> <biocase:value>Acinos arvensis (Lam.) Dandy</biocase:value> <biocase:value>Aconitum degenii ssp. paniculatum (Arcang.) Mucher</biocase:value> <biocase:value>Aconitum lycoctonum L. ssp. lycoctonum</biocase:value> <biocase:value>Aconitum napellus L. s.l.</biocase:value> ... </biocase:scan> </biocase:content> </syntaxhighlight> <strong>Search:</strong> For this type of request, the content consists of the ABCD dataset(s). See the the [[SampleABCDDocument| ABCD2 example]]. === Diagnostics (Debug Output) === This section is the most interesting part for debugging. If debug level is set to ''Info'' (which is recommended for debugging), it will document the processing of the request by the web service. The output is easy to understand, for example it will contain the SQL statements sent to the database. For a successful search request it could look similar to this: <syntaxhighlight> <biocase:diagnostics> <biocase:diagnostic severity="INFO"> Datasource wrapper FloraExsiccataBavarica requested </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Reading PSF from C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica \provider_setup_file.xml </biocase:diagnostic> <biocase:diagnostic severity="DEBUG"> PSF: PSF=C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\provider_setup_file.xml, recLimit=100, loglevel=20, user=webuser, database=rbg, dbIP=192.168.2.10, dbms=mysql, encoding=latin_1, schemas={u'http://www.tdwg.org/schemas/abcd/2.06': <biocase.wrapper.psf_handler.SupportedSchema instance at 0x011F5B98>}, tablegraph=GRAPH: graph: unit-metadata, +++ ALIAS2TABLE: {u'unit': u'feb3', u'metadata': u'metadata'} </biocase:diagnostic> <biocase:diagnostic severity="INFO"> BioCASe protocol used. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> HTTP parameter 'query' used for building the request. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT DISTINCT unit.id FROM feb3 AS unit WHERE (unit.nameautor LIKE 'Allium %' ) LIMIT 11' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT unit.id, metadata.supplier_organisation, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (40, 41, 42, 43, 44, 45, 46, 47)) ORDER BY unit.id' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> time to execute request is 0.46599984169 </biocase:diagnostic> </biocase:diagnostics> </syntaxhighlight> == Sample Errors and Warnings in the Debug Output == <syntaxhighlight> <biocase:diagnostic severity="ERROR"> No DB connection could be established. Please verify DB state and connection parameters in your provider_setup_file.xml. </biocase:diagnostic> </syntaxhighlight> Well... just what it says. The Provider Software could not connect to the database to be published. Check the database connection settings (server IP, database name, user, and password) and make sure you’ve installed the Python package for the DBMS you’re using (see the <small>Libs test</small> page in <small>Utilities</small>). <syntaxhighlight> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\flora\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="ERROR"> The CMF DB mapping file http://www.tdwg.org/schemas/abcd/2.06 was corrupt or could not be interpreted. </biocase:diagnostic> </syntaxhighlight> The requested schema (identified by the namespace http://www.tdwg.org/schemas/abcd/2.06) is not supported by that service. Create a mapping and map elements. <syntaxhighlight> <biocase:diagnostic severity="ERROR"> The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result /TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts </biocase:diagnostic> </syntaxhighlight> Obvious - happens if you try to use a concept that is not mapped in a scan request or a filter. <syntaxhighlight> <biocase:diagnostic severity="ERROR"> A SQL statement produced an error: SELECT unit.id, metadata.supplier_organisation, metadata.supplier_email, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) ORDER BY unit.id </biocase:diagnostic> </syntaxhighlight> That could have a number of reasons: One of the table/column names that were set up in the configuration does not exist (because it was renamed or removed) or the credentials used by the BPS do not have sufficient privileges. Simply copy the SQL statement and execute it manually on the database with a regular database client that will show you the detailed error message returned by the DBMS. <syntaxhighlight> <biocase:diagnostic severity="WARNING"> The mandatory child element SourceInstitutionID could not be created [0 existing, but 1 required]. The element /DataSets/DataSet/Units/Unit was also dropped. </biocase:diagnostic> <biocase:diagnostic severity="WARNING"> The mandatory child element Units could not be created [0 existing, but 1 required]. The element /DataSets/DataSet was also dropped. </biocase:diagnostic> </syntaxhighlight> One problem caused the next one in this sequence of warnings: The mandatory ABCD element ''SourceInstitutionID '' was either not mapped or empty; therefore the respective record (''unit'') was dropped. Because this happened for all units in the dataset, all units were dropped, resulting in an empty ''Units'' sub tree. The second warning tells you what happened then: Because the ''Units'' element is mandatory for the ''Dataset'' element, the whole dataset was dropped. As a result, the response ABCD document was empty. == Debugging a BioCASe Installation == In case the debug output of the web service doesn’t help you to find the problem, or if the BPS behaves not as intended without displaying any error messages, you should have a look at the debug logs written by the BPS. To turn on the Debug logs, go to the System Administration page and set <small>Debug logs</small> to <small>True</small>. [[File:instConfigTool.png|500px]] Redo the step that didn’t work and open the ''log'' folder of your BioCASe installation. There are several global log files (pywrapper_request.log, pywrapper_error.log, webapp_debug.log, webapp_error.log) and a debug files per data source (e.g. debug_flora.log). Open these files and see if they can give you a clue to the problem. If you still can’t get it running, [[Special:Contact|contact the BioCASe team]]. 23daaa234bc5ef6ea4396e69106e935d6e3d9175 359 358 2011-08-18T12:19:20Z JoergHoletschek 3 /* Content */ wikitext text/x-wiki This tutorial explains how to debug a BioCASe web service and a BioCASe installation in general. If you follow the instructions and don’t manage to get your service running, please [[Special:Contact|contact the BioCASe team]]. == Debugging a BioCASe Web Service == Debugging a BioCASe web service can be done with the manual Query Form. It allows you to load a BioCASe request (Search/Scan) from a template, edit the parameters of the request (e.g. the filter) and will show you the response document, which holds the requested ABCD records and useful debug information. === Opening the Query Form === There are several ways to open the Query Form. On the entry page of a datasource, <small>TEST and DEBUG</small> will open the Query Form: [[File:debugDatasourceEntryPage.png]] In the data source configuration tool, the link to the according query form can be always found in the link list at the top of the page. Moreover, in the mapping editor, the button <small>Test mapping!</small> will open the Query Form in a separate tab: [[File:abcdMappingEditorEmptyTop.png]] === Basic Usage === On top of the Query Form you can see the URL of your web service. In this case it is ''<nowiki>http://localhost/biocase/pywrapper.cgi?dsa=flora</nowiki>'', ''flora'' being the name of the datasource. Make sure this is the correct access point of the web service you want to debug! [[File:debugQueryformFull.png]] The usage of the Query Form is very straightforward: * Click on one of the links below the text box to load a template, for example for an ABCD2 Scan or ABCD2 Search; * Edit the request parameters; * Click on <small>Submit</small> to get the response document. The section <small>BioCASe protocol filter operators</small> lists the operators that can be used for creating complex filters that combine several filter criteria. More about creating filters can be found in the BioCASe Protocol documentation at http://www.biocase.org/products/protocols/index.shtml. Clicking on the <small>Show</small> link in the <small>Capabilities</small> section will show the Capabilities response in a frame of the Query Form. This is useful, since it shows the namespaces for the schemas supported by the web service (which is required in the request text) and XPaths of mapped ABCD elements (required for creating filters on these elements). By using Copy & Paste you can spare the trouble of entering these monstrous strings of characters. === Changing the Debug Level for a Data Source === Debug info can be set to one of four levels for a BioCASe data source: * <strong>Error</strong> is the highest level will only output errors that prevent the web server from answering a request. * <strong>Warning</strong> will output errors that keep BioCASe from answering a request and warnings that don’t prevent the fulfilling of the request, but should be analysed carefully. For example, they might tell you that records have been dropped because mandatory ABCD elements have not been mapped or were empty for certain records. * <strong>Info</strong> is the default and will add some more information about how the Provider Software processes the request. * <strong>Debug</strong> is the most detailed level and will produce very verbose debug output. It adds very comprehensive information about how the ABCD documents are constructed and is suitable only for experienced users. For most cases, keeping the default ''Info'' level should be sufficient for testing and debugging a web service. For production use, you should set it to ''Error'' later. You can do this in the configuration of a datasource under Settings: [[File:debugDatasourceSettings.png]] == Analyzing the Response Document == Upon clicking <small>Submit</small>, the Query Form will send the request to the web service and wait for the response. It might take some time – up to minutes – until the response document shows up. Amongst others, this depends on * the complexity of the query (especially the filters), * the size of the dataset returned, * the debug level set (“debug” results in very exhaustive debug output), * the performance of the servers running BioCASe and the database, * the network bandwidth. The response document consists of three sections (a full response document can be found [[SampleABCDDocument|here]]): [[File:debugResponseDocumentOverview.png]] === Header === This section contains information about the BioCASe installation – operating system, DBMS used, Python/BioCASe version etc. It is intended for a consumer application and won’t be very helpful for debugging: <syntaxhighlight> <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> </syntaxhighlight> === Content === This section makes up the payload of the response document, depending on the type of request: <strong>Capabilities:</strong> Lists all schemas supported by the web service. For each schema, all mapped concepts (identified by their XPaths) are listed with data type and searchable flag. <syntaxhighlight> <biocase:content recordCount="0" recordDropped="0" recordStart="0" totalSearchHits="0"> <biocase:capabilities> <biocase:SupportedSchemas namespace="http://www.tdwg.org/schemas/abcd/2.06" request="true" response="true"> <biocase:Concept datatype="normalizedString" searchable="1">/DataSets/DataSet/ContentContacts/ContentContact/Address</biocase:Concept> <biocase:Concept datatype="normalizedString" searchable="1">/DataSets/DataSet/ContentContacts/ContentContact/Email</biocase:Concept> <biocase:Concept datatype="normalizedString" searchable="1">/DataSets/DataSet/ContentContacts/ContentContact/Name</biocase:Concept> <biocase:Concept datatype="normalizedString" searchable="1">/DataSets/DataSet/Metadata/Description/Representation/Details</biocase:Concept> ... </biocase:SupportedSchemas> </biocase:capabilities> </biocase:content> </syntaxhighlight> <strong>Scan:</strong> The document will simply list all distinct values for the concept specified in the request. When done for the scientific name, it would look similar to this: <syntaxhighlight> <biocase:content recordCount="1664" recordDropped="0" recordStart="0" totalSearchHits="1664"> <biocase:scan> <biocase:value>Achillea atrata L.</biocase:value> <biocase:value>Achillea millefolium L.</biocase:value> <biocase:value>Achillea nobilis L. ssp. nobilis</biocase:value> <biocase:value>Achillea ptarmica L.</biocase:value> <biocase:value>Acinos alpinus (L.) Moench</biocase:value> <biocase:value>Acinos arvensis (Lam.) Dandy</biocase:value> <biocase:value>Aconitum degenii ssp. paniculatum (Arcang.) Mucher</biocase:value> <biocase:value>Aconitum lycoctonum L. ssp. lycoctonum</biocase:value> <biocase:value>Aconitum napellus L. s.l.</biocase:value> ... </biocase:scan> </biocase:content> </syntaxhighlight> <strong>Search:</strong> For this type of request, the content consists of the ABCD dataset(s). See the the [[SampleABCDDocument| ABCD2 example]]. === Diagnostics (Debug Output) === This section is the most interesting part for debugging. If debug level is set to ''Info'' (which is recommended for debugging), it will document the processing of the request by the web service. The output is easy to understand, for example it will contain the SQL statements sent to the database. For a successful search request it could look similar to this: <syntaxhighlight> <biocase:diagnostics> <biocase:diagnostic severity="INFO"> Datasource wrapper FloraExsiccataBavarica requested </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Reading PSF from C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica \provider_setup_file.xml </biocase:diagnostic> <biocase:diagnostic severity="DEBUG"> PSF: PSF=C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\provider_setup_file.xml, recLimit=100, loglevel=20, user=webuser, database=rbg, dbIP=192.168.2.10, dbms=mysql, encoding=latin_1, schemas={u'http://www.tdwg.org/schemas/abcd/2.06': <biocase.wrapper.psf_handler.SupportedSchema instance at 0x011F5B98>}, tablegraph=GRAPH: graph: unit-metadata, +++ ALIAS2TABLE: {u'unit': u'feb3', u'metadata': u'metadata'} </biocase:diagnostic> <biocase:diagnostic severity="INFO"> BioCASe protocol used. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> HTTP parameter 'query' used for building the request. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT DISTINCT unit.id FROM feb3 AS unit WHERE (unit.nameautor LIKE 'Allium %' ) LIMIT 11' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT unit.id, metadata.supplier_organisation, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (40, 41, 42, 43, 44, 45, 46, 47)) ORDER BY unit.id' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> time to execute request is 0.46599984169 </biocase:diagnostic> </biocase:diagnostics> </syntaxhighlight> == Sample Errors and Warnings in the Debug Output == <syntaxhighlight> <biocase:diagnostic severity="ERROR"> No DB connection could be established. Please verify DB state and connection parameters in your provider_setup_file.xml. </biocase:diagnostic> </syntaxhighlight> Well... just what it says. The Provider Software could not connect to the database to be published. Check the database connection settings (server IP, database name, user, and password) and make sure you’ve installed the Python package for the DBMS you’re using (see the <small>Libs test</small> page in <small>Utilities</small>). <syntaxhighlight> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\flora\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="ERROR"> The CMF DB mapping file http://www.tdwg.org/schemas/abcd/2.06 was corrupt or could not be interpreted. </biocase:diagnostic> </syntaxhighlight> The requested schema (identified by the namespace http://www.tdwg.org/schemas/abcd/2.06) is not supported by that service. Create a mapping and map elements. <syntaxhighlight> <biocase:diagnostic severity="ERROR"> The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result /TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts </biocase:diagnostic> </syntaxhighlight> Obvious - happens if you try to use a concept that is not mapped in a scan request or a filter. <syntaxhighlight> <biocase:diagnostic severity="ERROR"> A SQL statement produced an error: SELECT unit.id, metadata.supplier_organisation, metadata.supplier_email, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) ORDER BY unit.id </biocase:diagnostic> </syntaxhighlight> That could have a number of reasons: One of the table/column names that were set up in the configuration does not exist (because it was renamed or removed) or the credentials used by the BPS do not have sufficient privileges. Simply copy the SQL statement and execute it manually on the database with a regular database client that will show you the detailed error message returned by the DBMS. <syntaxhighlight> <biocase:diagnostic severity="WARNING"> The mandatory child element SourceInstitutionID could not be created [0 existing, but 1 required]. The element /DataSets/DataSet/Units/Unit was also dropped. </biocase:diagnostic> <biocase:diagnostic severity="WARNING"> The mandatory child element Units could not be created [0 existing, but 1 required]. The element /DataSets/DataSet was also dropped. </biocase:diagnostic> </syntaxhighlight> One problem caused the next one in this sequence of warnings: The mandatory ABCD element ''SourceInstitutionID '' was either not mapped or empty; therefore the respective record (''unit'') was dropped. Because this happened for all units in the dataset, all units were dropped, resulting in an empty ''Units'' sub tree. The second warning tells you what happened then: Because the ''Units'' element is mandatory for the ''Dataset'' element, the whole dataset was dropped. As a result, the response ABCD document was empty. == Debugging a BioCASe Installation == In case the debug output of the web service doesn’t help you to find the problem, or if the BPS behaves not as intended without displaying any error messages, you should have a look at the debug logs written by the BPS. To turn on the Debug logs, go to the System Administration page and set <small>Debug logs</small> to <small>True</small>. [[File:instConfigTool.png|500px]] Redo the step that didn’t work and open the ''log'' folder of your BioCASe installation. There are several global log files (pywrapper_request.log, pywrapper_error.log, webapp_debug.log, webapp_error.log) and a debug files per data source (e.g. debug_flora.log). Open these files and see if they can give you a clue to the problem. If you still can’t get it running, [[Special:Contact|contact the BioCASe team]]. c5fea9c9f82078ab9aad9d3601577ad28bd99d6c 360 359 2011-08-18T12:20:59Z JoergHoletschek 3 /* Sample Errors and Warnings in the Debug Output */ wikitext text/x-wiki This tutorial explains how to debug a BioCASe web service and a BioCASe installation in general. If you follow the instructions and don’t manage to get your service running, please [[Special:Contact|contact the BioCASe team]]. == Debugging a BioCASe Web Service == Debugging a BioCASe web service can be done with the manual Query Form. It allows you to load a BioCASe request (Search/Scan) from a template, edit the parameters of the request (e.g. the filter) and will show you the response document, which holds the requested ABCD records and useful debug information. === Opening the Query Form === There are several ways to open the Query Form. On the entry page of a datasource, <small>TEST and DEBUG</small> will open the Query Form: [[File:debugDatasourceEntryPage.png]] In the data source configuration tool, the link to the according query form can be always found in the link list at the top of the page. Moreover, in the mapping editor, the button <small>Test mapping!</small> will open the Query Form in a separate tab: [[File:abcdMappingEditorEmptyTop.png]] === Basic Usage === On top of the Query Form you can see the URL of your web service. In this case it is ''<nowiki>http://localhost/biocase/pywrapper.cgi?dsa=flora</nowiki>'', ''flora'' being the name of the datasource. Make sure this is the correct access point of the web service you want to debug! [[File:debugQueryformFull.png]] The usage of the Query Form is very straightforward: * Click on one of the links below the text box to load a template, for example for an ABCD2 Scan or ABCD2 Search; * Edit the request parameters; * Click on <small>Submit</small> to get the response document. The section <small>BioCASe protocol filter operators</small> lists the operators that can be used for creating complex filters that combine several filter criteria. More about creating filters can be found in the BioCASe Protocol documentation at http://www.biocase.org/products/protocols/index.shtml. Clicking on the <small>Show</small> link in the <small>Capabilities</small> section will show the Capabilities response in a frame of the Query Form. This is useful, since it shows the namespaces for the schemas supported by the web service (which is required in the request text) and XPaths of mapped ABCD elements (required for creating filters on these elements). By using Copy & Paste you can spare the trouble of entering these monstrous strings of characters. === Changing the Debug Level for a Data Source === Debug info can be set to one of four levels for a BioCASe data source: * <strong>Error</strong> is the highest level will only output errors that prevent the web server from answering a request. * <strong>Warning</strong> will output errors that keep BioCASe from answering a request and warnings that don’t prevent the fulfilling of the request, but should be analysed carefully. For example, they might tell you that records have been dropped because mandatory ABCD elements have not been mapped or were empty for certain records. * <strong>Info</strong> is the default and will add some more information about how the Provider Software processes the request. * <strong>Debug</strong> is the most detailed level and will produce very verbose debug output. It adds very comprehensive information about how the ABCD documents are constructed and is suitable only for experienced users. For most cases, keeping the default ''Info'' level should be sufficient for testing and debugging a web service. For production use, you should set it to ''Error'' later. You can do this in the configuration of a datasource under Settings: [[File:debugDatasourceSettings.png]] == Analyzing the Response Document == Upon clicking <small>Submit</small>, the Query Form will send the request to the web service and wait for the response. It might take some time – up to minutes – until the response document shows up. Amongst others, this depends on * the complexity of the query (especially the filters), * the size of the dataset returned, * the debug level set (“debug” results in very exhaustive debug output), * the performance of the servers running BioCASe and the database, * the network bandwidth. The response document consists of three sections (a full response document can be found [[SampleABCDDocument|here]]): [[File:debugResponseDocumentOverview.png]] === Header === This section contains information about the BioCASe installation – operating system, DBMS used, Python/BioCASe version etc. It is intended for a consumer application and won’t be very helpful for debugging: <syntaxhighlight> <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> </syntaxhighlight> === Content === This section makes up the payload of the response document, depending on the type of request: <strong>Capabilities:</strong> Lists all schemas supported by the web service. For each schema, all mapped concepts (identified by their XPaths) are listed with data type and searchable flag. <syntaxhighlight> <biocase:content recordCount="0" recordDropped="0" recordStart="0" totalSearchHits="0"> <biocase:capabilities> <biocase:SupportedSchemas namespace="http://www.tdwg.org/schemas/abcd/2.06" request="true" response="true"> <biocase:Concept datatype="normalizedString" searchable="1">/DataSets/DataSet/ContentContacts/ContentContact/Address</biocase:Concept> <biocase:Concept datatype="normalizedString" searchable="1">/DataSets/DataSet/ContentContacts/ContentContact/Email</biocase:Concept> <biocase:Concept datatype="normalizedString" searchable="1">/DataSets/DataSet/ContentContacts/ContentContact/Name</biocase:Concept> <biocase:Concept datatype="normalizedString" searchable="1">/DataSets/DataSet/Metadata/Description/Representation/Details</biocase:Concept> ... </biocase:SupportedSchemas> </biocase:capabilities> </biocase:content> </syntaxhighlight> <strong>Scan:</strong> The document will simply list all distinct values for the concept specified in the request. When done for the scientific name, it would look similar to this: <syntaxhighlight> <biocase:content recordCount="1664" recordDropped="0" recordStart="0" totalSearchHits="1664"> <biocase:scan> <biocase:value>Achillea atrata L.</biocase:value> <biocase:value>Achillea millefolium L.</biocase:value> <biocase:value>Achillea nobilis L. ssp. nobilis</biocase:value> <biocase:value>Achillea ptarmica L.</biocase:value> <biocase:value>Acinos alpinus (L.) Moench</biocase:value> <biocase:value>Acinos arvensis (Lam.) Dandy</biocase:value> <biocase:value>Aconitum degenii ssp. paniculatum (Arcang.) Mucher</biocase:value> <biocase:value>Aconitum lycoctonum L. ssp. lycoctonum</biocase:value> <biocase:value>Aconitum napellus L. s.l.</biocase:value> ... </biocase:scan> </biocase:content> </syntaxhighlight> <strong>Search:</strong> For this type of request, the content consists of the ABCD dataset(s). See the the [[SampleABCDDocument| ABCD2 example]]. === Diagnostics (Debug Output) === This section is the most interesting part for debugging. If debug level is set to ''Info'' (which is recommended for debugging), it will document the processing of the request by the web service. The output is easy to understand, for example it will contain the SQL statements sent to the database. For a successful search request it could look similar to this: <syntaxhighlight> <biocase:diagnostics> <biocase:diagnostic severity="INFO"> Datasource wrapper FloraExsiccataBavarica requested </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Reading PSF from C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica \provider_setup_file.xml </biocase:diagnostic> <biocase:diagnostic severity="DEBUG"> PSF: PSF=C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\provider_setup_file.xml, recLimit=100, loglevel=20, user=webuser, database=rbg, dbIP=192.168.2.10, dbms=mysql, encoding=latin_1, schemas={u'http://www.tdwg.org/schemas/abcd/2.06': <biocase.wrapper.psf_handler.SupportedSchema instance at 0x011F5B98>}, tablegraph=GRAPH: graph: unit-metadata, +++ ALIAS2TABLE: {u'unit': u'feb3', u'metadata': u'metadata'} </biocase:diagnostic> <biocase:diagnostic severity="INFO"> BioCASe protocol used. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> HTTP parameter 'query' used for building the request. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT DISTINCT unit.id FROM feb3 AS unit WHERE (unit.nameautor LIKE 'Allium %' ) LIMIT 11' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT unit.id, metadata.supplier_organisation, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (40, 41, 42, 43, 44, 45, 46, 47)) ORDER BY unit.id' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> time to execute request is 0.46599984169 </biocase:diagnostic> </biocase:diagnostics> </syntaxhighlight> == Sample Errors and Warnings in the Debug Output == <syntaxhighlight> <biocase:diagnostic severity="ERROR"> No DB connection could be established. Please verify DB state and connection parameters in your provider_setup_file.xml. </biocase:diagnostic> </syntaxhighlight> Well... just what it says. The Provider Software could not connect to the database to be published. Check the database connection settings (server IP, database name, user, and password) and make sure you’ve installed the Python package for the DBMS you’re using (see the <small>Libs test</small> page in <small>Utilities</small>). <syntaxhighlight> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\flora\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="ERROR"> The CMF DB mapping file http://www.tdwg.org/schemas/abcd/2.06 was corrupt or could not be interpreted. </biocase:diagnostic> </syntaxhighlight> The requested schema (identified by the namespace http://www.tdwg.org/schemas/abcd/2.06) is not supported by that service. Create a mapping and map elements. <syntaxhighlight> <biocase:diagnostic severity="ERROR"> The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result /TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts </biocase:diagnostic> </syntaxhighlight> Obvious - happens if you try to use a concept that is not mapped in a scan request or a filter. <syntaxhighlight> <biocase:diagnostic severity="ERROR"> A SQL statement produced an error: SELECT unit.id, metadata.supplier_organisation, metadata.supplier_email, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) ORDER BY unit.id </biocase:diagnostic> </syntaxhighlight> That could have a number of reasons: One of the table/column names that were set up in the configuration does not exist (because it was renamed or removed) or the credentials used by the BPS do not have sufficient privileges. Simply copy the SQL statement and execute it manually on the database with a regular database client that will show you the detailed error message returned by the DBMS. <syntaxhighlight> <biocase:diagnostic severity="WARNING"> The mandatory child element SourceInstitutionID could not be created [0 existing, but 1 required]. The element /DataSets/DataSet/Units/Unit was also dropped. </biocase:diagnostic> <biocase:diagnostic severity="WARNING"> The mandatory child element Units could not be created [0 existing, but 1 required]. The element /DataSets/DataSet was also dropped. </biocase:diagnostic> </syntaxhighlight> One problem caused the next one in this sequence of warnings: The mandatory ABCD element ''SourceInstitutionID '' was either not mapped or empty; therefore the respective record (''unit'') was dropped. Because this happened for all units in the dataset, all units were dropped, resulting in an empty ''Units'' sub tree. The second warning tells you what happened then: Because the ''Units'' element is mandatory for the ''Dataset'' element, the whole dataset was dropped. As a result, the response ABCD document was empty. == Debugging a BioCASe Installation == In case the debug output of the web service doesn’t help you to find the problem, or if the BPS behaves not as intended without displaying any error messages, you should have a look at the debug logs written by the BPS. To turn on the Debug logs, go to the System Administration page and set <small>Debug logs</small> to <small>True</small>. [[File:instConfigTool.png|500px]] Redo the step that didn’t work and open the ''log'' folder of your BioCASe installation. There are several global log files (pywrapper_request.log, pywrapper_error.log, webapp_debug.log, webapp_error.log) and a debug files per data source (e.g. debug_flora.log). Open these files and see if they can give you a clue to the problem. If you still can’t get it running, [[Special:Contact|contact the BioCASe team]]. 96f164cd83d345909e46c6e1622ac420356d9c07 361 360 2011-08-18T12:25:39Z JoergHoletschek 3 /* Sample Errors and Warnings in the Debug Output */ wikitext text/x-wiki This tutorial explains how to debug a BioCASe web service and a BioCASe installation in general. If you follow the instructions and don’t manage to get your service running, please [[Special:Contact|contact the BioCASe team]]. == Debugging a BioCASe Web Service == Debugging a BioCASe web service can be done with the manual Query Form. It allows you to load a BioCASe request (Search/Scan) from a template, edit the parameters of the request (e.g. the filter) and will show you the response document, which holds the requested ABCD records and useful debug information. === Opening the Query Form === There are several ways to open the Query Form. On the entry page of a datasource, <small>TEST and DEBUG</small> will open the Query Form: [[File:debugDatasourceEntryPage.png]] In the data source configuration tool, the link to the according query form can be always found in the link list at the top of the page. Moreover, in the mapping editor, the button <small>Test mapping!</small> will open the Query Form in a separate tab: [[File:abcdMappingEditorEmptyTop.png]] === Basic Usage === On top of the Query Form you can see the URL of your web service. In this case it is ''<nowiki>http://localhost/biocase/pywrapper.cgi?dsa=flora</nowiki>'', ''flora'' being the name of the datasource. Make sure this is the correct access point of the web service you want to debug! [[File:debugQueryformFull.png]] The usage of the Query Form is very straightforward: * Click on one of the links below the text box to load a template, for example for an ABCD2 Scan or ABCD2 Search; * Edit the request parameters; * Click on <small>Submit</small> to get the response document. The section <small>BioCASe protocol filter operators</small> lists the operators that can be used for creating complex filters that combine several filter criteria. More about creating filters can be found in the BioCASe Protocol documentation at http://www.biocase.org/products/protocols/index.shtml. Clicking on the <small>Show</small> link in the <small>Capabilities</small> section will show the Capabilities response in a frame of the Query Form. This is useful, since it shows the namespaces for the schemas supported by the web service (which is required in the request text) and XPaths of mapped ABCD elements (required for creating filters on these elements). By using Copy & Paste you can spare the trouble of entering these monstrous strings of characters. === Changing the Debug Level for a Data Source === Debug info can be set to one of four levels for a BioCASe data source: * <strong>Error</strong> is the highest level will only output errors that prevent the web server from answering a request. * <strong>Warning</strong> will output errors that keep BioCASe from answering a request and warnings that don’t prevent the fulfilling of the request, but should be analysed carefully. For example, they might tell you that records have been dropped because mandatory ABCD elements have not been mapped or were empty for certain records. * <strong>Info</strong> is the default and will add some more information about how the Provider Software processes the request. * <strong>Debug</strong> is the most detailed level and will produce very verbose debug output. It adds very comprehensive information about how the ABCD documents are constructed and is suitable only for experienced users. For most cases, keeping the default ''Info'' level should be sufficient for testing and debugging a web service. For production use, you should set it to ''Error'' later. You can do this in the configuration of a datasource under Settings: [[File:debugDatasourceSettings.png]] == Analyzing the Response Document == Upon clicking <small>Submit</small>, the Query Form will send the request to the web service and wait for the response. It might take some time – up to minutes – until the response document shows up. Amongst others, this depends on * the complexity of the query (especially the filters), * the size of the dataset returned, * the debug level set (“debug” results in very exhaustive debug output), * the performance of the servers running BioCASe and the database, * the network bandwidth. The response document consists of three sections (a full response document can be found [[SampleABCDDocument|here]]): [[File:debugResponseDocumentOverview.png]] === Header === This section contains information about the BioCASe installation – operating system, DBMS used, Python/BioCASe version etc. It is intended for a consumer application and won’t be very helpful for debugging: <syntaxhighlight> <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> </syntaxhighlight> === Content === This section makes up the payload of the response document, depending on the type of request: <strong>Capabilities:</strong> Lists all schemas supported by the web service. For each schema, all mapped concepts (identified by their XPaths) are listed with data type and searchable flag. <syntaxhighlight> <biocase:content recordCount="0" recordDropped="0" recordStart="0" totalSearchHits="0"> <biocase:capabilities> <biocase:SupportedSchemas namespace="http://www.tdwg.org/schemas/abcd/2.06" request="true" response="true"> <biocase:Concept datatype="normalizedString" searchable="1">/DataSets/DataSet/ContentContacts/ContentContact/Address</biocase:Concept> <biocase:Concept datatype="normalizedString" searchable="1">/DataSets/DataSet/ContentContacts/ContentContact/Email</biocase:Concept> <biocase:Concept datatype="normalizedString" searchable="1">/DataSets/DataSet/ContentContacts/ContentContact/Name</biocase:Concept> <biocase:Concept datatype="normalizedString" searchable="1">/DataSets/DataSet/Metadata/Description/Representation/Details</biocase:Concept> ... </biocase:SupportedSchemas> </biocase:capabilities> </biocase:content> </syntaxhighlight> <strong>Scan:</strong> The document will simply list all distinct values for the concept specified in the request. When done for the scientific name, it would look similar to this: <syntaxhighlight> <biocase:content recordCount="1664" recordDropped="0" recordStart="0" totalSearchHits="1664"> <biocase:scan> <biocase:value>Achillea atrata L.</biocase:value> <biocase:value>Achillea millefolium L.</biocase:value> <biocase:value>Achillea nobilis L. ssp. nobilis</biocase:value> <biocase:value>Achillea ptarmica L.</biocase:value> <biocase:value>Acinos alpinus (L.) Moench</biocase:value> <biocase:value>Acinos arvensis (Lam.) Dandy</biocase:value> <biocase:value>Aconitum degenii ssp. paniculatum (Arcang.) Mucher</biocase:value> <biocase:value>Aconitum lycoctonum L. ssp. lycoctonum</biocase:value> <biocase:value>Aconitum napellus L. s.l.</biocase:value> ... </biocase:scan> </biocase:content> </syntaxhighlight> <strong>Search:</strong> For this type of request, the content consists of the ABCD dataset(s). See the the [[SampleABCDDocument| ABCD2 example]]. === Diagnostics (Debug Output) === This section is the most interesting part for debugging. If debug level is set to ''Info'' (which is recommended for debugging), it will document the processing of the request by the web service. The output is easy to understand, for example it will contain the SQL statements sent to the database. For a successful search request it could look similar to this: <syntaxhighlight> <biocase:diagnostics> <biocase:diagnostic severity="INFO"> Datasource wrapper FloraExsiccataBavarica requested </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Reading PSF from C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica \provider_setup_file.xml </biocase:diagnostic> <biocase:diagnostic severity="DEBUG"> PSF: PSF=C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\provider_setup_file.xml, recLimit=100, loglevel=20, user=webuser, database=rbg, dbIP=192.168.2.10, dbms=mysql, encoding=latin_1, schemas={u'http://www.tdwg.org/schemas/abcd/2.06': <biocase.wrapper.psf_handler.SupportedSchema instance at 0x011F5B98>}, tablegraph=GRAPH: graph: unit-metadata, +++ ALIAS2TABLE: {u'unit': u'feb3', u'metadata': u'metadata'} </biocase:diagnostic> <biocase:diagnostic severity="INFO"> BioCASe protocol used. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> HTTP parameter 'query' used for building the request. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT DISTINCT unit.id FROM feb3 AS unit WHERE (unit.nameautor LIKE 'Allium %' ) LIMIT 11' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT unit.id, metadata.supplier_organisation, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (40, 41, 42, 43, 44, 45, 46, 47)) ORDER BY unit.id' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> time to execute request is 0.46599984169 </biocase:diagnostic> </biocase:diagnostics> </syntaxhighlight> == Sample Errors and Warnings in the Debug Output == <syntaxhighlight> <biocase:diagnostic severity="ERROR"> No DB connection could be established. Please verify DB state and connection parameters in your provider_setup_file.xml. </biocase:diagnostic> </syntaxhighlight> Well... just what it says. The Provider Software could not connect to the database to be published. Check the database connection settings (server IP, database name, user, and password) and make sure you’ve installed the Python package for the DBMS you’re using (see the <small>Libs test</small> page in <small>Utilities</small>). <syntaxhighlight> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\flora\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="ERROR"> The CMF DB mapping file http://www.tdwg.org/schemas/abcd/2.06 was corrupt or could not be interpreted. </biocase:diagnostic> </syntaxhighlight> The requested schema (identified by the namespace http://www.tdwg.org/schemas/abcd/2.06) is not supported by that service. Create a mapping for this schema and map elements. <syntaxhighlight> <biocase:diagnostic severity="ERROR"> The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result /TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts </biocase:diagnostic> </syntaxhighlight> Obvious - happens if you try to use a concept that is not mapped in a scan request or a filter. <syntaxhighlight> <biocase:diagnostic severity="ERROR"> A SQL statement produced an error: SELECT unit.id, metadata.supplier_organisation, metadata.supplier_email, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) ORDER BY unit.id </biocase:diagnostic> </syntaxhighlight> That could have a number of reasons: One of the table/column names that were set up in the configuration does not exist (because it was renamed or removed) or the credentials used by the BPS do not have sufficient privileges. Simply copy the SQL statement and execute it manually on the database with a regular database client that will show you the detailed error message returned by the DBMS. <syntaxhighlight> <biocase:diagnostic severity="WARNING"> The mandatory child element SourceInstitutionID could not be created [0 existing, but 1 required]. The element /DataSets/DataSet/Units/Unit was also dropped. </biocase:diagnostic> <biocase:diagnostic severity="WARNING"> The mandatory child element Units could not be created [0 existing, but 1 required]. The element /DataSets/DataSet was also dropped. </biocase:diagnostic> </syntaxhighlight> One problem caused the next one in this sequence of warnings: The mandatory ABCD element ''SourceInstitutionID '' was either not mapped or empty; therefore the respective record (''unit'') was dropped. Because this happened for all units in the dataset, all units were dropped, resulting in an empty ''Units'' sub tree. The second warning tells you what happened then: Because the ''Units'' element is mandatory for the ''Dataset'' element, the whole dataset was dropped. As a result, the response ABCD document was empty. == Debugging a BioCASe Installation == In case the debug output of the web service doesn’t help you to find the problem, or if the BPS behaves not as intended without displaying any error messages, you should have a look at the debug logs written by the BPS. To turn on the Debug logs, go to the System Administration page and set <small>Debug logs</small> to <small>True</small>. [[File:instConfigTool.png|500px]] Redo the step that didn’t work and open the ''log'' folder of your BioCASe installation. There are several global log files (pywrapper_request.log, pywrapper_error.log, webapp_debug.log, webapp_error.log) and a debug files per data source (e.g. debug_flora.log). Open these files and see if they can give you a clue to the problem. If you still can’t get it running, [[Special:Contact|contact the BioCASe team]]. 92e30762bf4d51402f91e36d49d9605d71b54d11 0 29 362 355 2011-08-18T13:39:58Z JoergHoletschek 3 /* Adding additional ABCD Elements */ wikitext text/x-wiki This tutorial will explain how to set up a mapping for the ABCD2.06 data schema. If you intend to map another schema (e.g. one of the ABCD extension ABCD-EFG, ABCD-DNA or HISPID), you still should take the time to read it. Large portions of the process will be identical. == Access to Biological Collection Data Schema (ABCD) == ABCD in its current version 2.06 is the data schema typically used in conjunction with BioCASe. It is a highly complex XML data schema with about 1,000 elements that is able to store almost every piece of information that can be found in a natural history specimen collection or an observation database. The complexity is due to the fact that it can be used for a wide range of collections/databases – for living and preserved specimens, observations and culture collections, for zoological, botanical, bacterial and viral collections, marine or terrestrial, for herbaria, botanic and zoological gardens. For each of these special types, ABCD features special sections where information specific for this type can be stored. Other sections of ABCD will be shared by all these types, for example for gathering site, identifier or metadata. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a list of commonly used ABCD elements on the Wiki page [[CommonABCD2Concepts|Common ABCD2 Concepts]]. == Creating an ABCD Mapping for a Data Source == If you created a data source from a template (e.g. ''abcdmetadata''), a schema mapping for ABCD will be already existent. In this case, you can directly jump to the next section. If you created an empty data source, the section <small>Schemas</small> in the data source configuration overview will be empty: [[File:abcdSchemaMappings.png]] In order to create a new schema mapping, select the desired schema from the list and click <small>Create</small>. For ABCD, choose <small>ABCD_2.06.xml</small> (be careful not to hit one of the ABCD extensions, they look quite similar). After clicking <small>Create</small>, you will be directed to the mapping editor, which will allow you to edit the mappings. == The Mapping Editor == The mapping editor allows you to add, remove and edit ABCD mappings. On top you’ll find some summary information, on the bottom you’ll see the structure of the ABCD mapping: [[File:abcdMappingEditorEmpty.png]] First thing we will do is to set the root table, which is the table that holds the records you want to publish (specimens or observations). That table must hold one record per object to be published, identified by a unique identifier (called ''UnitID''). This identifier doesn't need to be the primary key necessarily, as long as it is unique (if that sentence confuses you, ignore it and just keep in mind that the UnitID must be unique.) Use the drop down box labelled <small>Root table Alias</small> to select the root table of your data model. If your metadata table holds just one record (because all occurrences published share the same metadata) and if it is not linked to the root table by a foreign key, you can specify this table in the drop-down box <small>Static table Alias</small>. The Provider Software will use a natural join to connect the metadata record stored in this table to all occurrences published. If your metadata table is referenced by a foreign key in the root able, you must not declare the metadata table as a static table! So either join the metadata table using a foreign key OR declare it as a static table. When you’ve set the root table (and static table, if you use that), press <small>Save</small>. As with the DB structure setup, this will write your changes to the configuration files, closing the tab without saving will discard your changes! Also, in case you messed something up, you can use the <small>Revert</small> button to restore the state when you’ve saved for the last time. In the mapping editor, the link <small>Overview</small> at the top of the page will take you back to the datasource configuration overview. After creating a mapping, it will appear in the section <small>Schemas</small>. The entry will show the number of mapped elements (0 for now) and the schema namespace. Pressing the trash can symbol will move it to the trash can, which means that your web service does not support this schema anymore; if you did this by mistake, you can restore it by pressing the <small>Restore</small> button. <small>Purge</small> will delete the schema mapping permanently: [[File:abcdSchemaMappings2.png]] You can map several schemas for a given datasource. For example, in addition to the current ABCD2 standard, you could add the deprecated version 1.2 (even though I have no idea why you should do that) or DarwinCore. Just select the appropriate schema in the drop-down list and click <small>Create</small>. Since the namespace of a schema will be used in requests, you cannot create two mappings for the same namespace. Trying this will get you an error message. == Adding Mandatory ABCD Elements == Clicking on a schema in the datasource overview page will take you again to the mapping editor. In the lower half of the page, you will see the basic structure of an ABCD document: [[File:abcdMappingEditorEmptyBottom.png|border]] <strong>Technical/Content Contact:</strong> These trees hold information on who should be contacted for technical issues or questions concerning the published database, each with name, address, email address and telephone number. <strong>Metadata:</strong> In this sub tree you’ll find all metadata fields, for example title and description of the dataset, ownership information, date last modified, IPR statements etc. <strong>Units:</strong> This part stores the "real" data you want to publish, that means all specimen or observation data. The three elements already shown in the tree are identifiers that make up the so called "Triple ID", which uniquely identifies records within primary biodiversity networks such as BioCASe and GBIF: * ''UnitID'', which is a unique identifier for each record to be published (you can use the catalogue number, for example, or an existing barcode); * ''SourceInstitutionID'', an identifier for the institution publishing the dataset (for example BGBM); * ''SourceID'', an identifier for the dataset (e.g. “Herbarium Berolinense’’). The elements already shown in the mapping editor are printed in red because they are mandatory, which means they are required for an ABCD document. Without them being mapped (to database fields that are non-empty), the Provider Software will not be able to construct valid ABCD documents and will not publish any records. So the first elements that should be mapped are the mandatory fields. To add a mapping, click the <strong>+</strong> symbol next to the field, which will open the mapping editor dialog, where you can select the database table/column you want to map the ABCD element to: [[File:abcdMappingEditorDialog.png]] In this case, the element ''/DataSets/DataSet/Metadata/RevisionData/DateModified'' is mapped to the column ''source_update'' of the ''metadata'' table, which has the data type ''date''. (Remember that table/column retrieval is not supported for all DBMS, so if you’re unlucky, you might need to type in table/column name manually.) <small>Save</small> will accept your settings and close the dialog, <small>Cancel</small> will discard any changes. Instead of mapping an element to a database field, you can map it to a literal. For example, you could enter the technical and content contact information directly as literals in the mapping editor dialog (assuming they’re the same for all records). However, we strongly discourage you from doing this. For one thing, this is just bad style, since it will scatter the information published over different places: Some will be retrieved from the database, some (the literals) will be loaded from the configuration files. Changing the metadata of your service later would then require access to the database and access to the BioCASe configuration – that means the configuration tool password. Be nice to the person who will do these changes and put everything neatly in a database table! The second reason is even more important: Using literals for an element will result in this element to become not searchable, which means it cannot be used in a filter of a Search or a Scan request. Even though this behaviour is discoverable with a Capabilities request (by evaluating the ''searchable'' flag), a potential user of your web service might just not think of that, so it’s a good idea to avoid that in first place. Please use literals sparsely! In the mapping editor window, you can select several source columns and several literals, even a combination of both. The Provider Software will simply concatenate the values retrieved from the database and the literals specified (in the order they’re given). But again: Keep in mind that this will make the element not searchable. A better solution is to do any concatenation operations in a database view. Use the mapping editor window to add mappings for all eight mandatory fields: Click on the <strong>+</strong> symbol next to the element, choose table/column name and the correct data type, then press <small>Save</small>. When you’re done, the page should look similar to the following image. Press <small>Save</small> in the main window to write your changes to the configuration files. [[File:abcdMappingEditorMandatoryElements.png]] == Initial Testing of the Data Source == Once you’ve mapped all mandatory elements, you can do the first test of your BioCASe web service. Even though you haven’t mapped any real information yet, it is a good idea to check the service and solve any problems hitherto before continuing with the mapping. Click on <small>Test mapping!</small> to open the Query Form that can be used for this purpose: [[File:abcdQueryFormEmpty.png]] With the Query Form you can send a BioCASe request to your web service; it will wait for the reply of the web service and display the response document. On top of the page you can see the URL of your web service. In this case it is ''<nowiki>http://localhost/biocase/pywrapper.cgi?dsa=flora</nowiki>'', ''flora'' being the name of the datasource. Make sure this is correct for your installation! Into the text box below you can enter the request XML text. Because it would be cumbersome to enter the whole BioCASe request, you can use the links below the box to load templates. For now, click on <small>ABCD2 Search</small> to fill the box with a BioCASe Search request for documents in the ABCD2 format (the one you’ve just mapped): <syntaxhighlight> <?xml version='1.0' encoding='UTF-8'?> <request xmlns='http://www.biocase.org/schemas/protocol/1.3'> <header><type>search</type></header> <search> <requestFormat>http://www.tdwg.org/schemas/abcd/2.06</requestFormat> <responseFormat start='0' limit='10'>http://www.tdwg.org/schemas/abcd/2.06</responseFormat> <filter> <like path='/DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString'>A*</like> </filter> <count>false</count> </search> </request> </syntaxhighlight> As you can see, there is a filter set on the ''ScientificName'' element. Since we haven’t mapped this element yet, this won’t work. So let’s remove the filter (all lines between <filter> and </filter>, including the ''filter'' tags), leaving the text box like this: [[File:abcdQueryFormFilled.png]] Pressing <small>Submit</small> will send the request to the web service and – hopefully – display the response document with the correct ABCD records. For the mapping we’ve just created, it should look similar to the following (use the "–" symbol next to the trees TechnicalContact, ContentContact and Metadata to collapse them): [[File:abcdResponseMandatoryElements.png]] If you get an error, please check out the [[Debugging]] tutorial on how to debug a web service. You should continue the mapping process only after your web service returns results for the basal mapping we’ve just created. == Adding additional ABCD Elements == Once you’ve successfully managed to map the mandatory ABCD elements ("successfully" means the web service returns ABCD documents), you can go on with the mapping process. This is usually an iterative process: You add a handful of elements (for example, country, country code, locality text and coordinates for the gathering site), then you use the Query Form to see if the information ends up where it is supposed to do. This is the reason the Query Form opens up in a separate tab (or window, of you configured your web browser so): In one tab, you use the mapping editor to add new elements; after pressing <small>Save</small> you go to the Query Form tab and see what the response documents look like. Redo this until you’ve mapped all information you want to publish. In order to map new ABCD concepts, tick the check box labelled <small>Show all concepts</small> and press <small>Refresh</small>. The mapping editor will then display the whole ABCD tree, which looks admittedly a bit intimidating. In order to find the concept you’re looking for, use the search function of your browser. If you want to map the country name of the gathering site, for example, hit <small>Ctrl + F</small> (for Firefox, Microsoft IE and Opera) and type in the name of the ABCD element, namely ''country''. The browser will jump to the ''Country'' tree of the ''Gathering'' element, where you’ll find the elements for country name (''Name''), ISO country code (''ISO3166Code'') and the country name in the country’s language (''NameDerived''): [[File:abcdMappingEditorSearch.png|500px]] Continue the sequence mapping new elements/testing until you’ve mapped all information you want to publish. You can have a look at the list of recommend and commonly used ABCD elements on the [[CommonABCD2Concepts|Common ABCD2 Concepts]] page. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki]. The following page will show you a sample response ABCD document for a web service with 77 concepts mapped. Of course your documents will look different, because you will probably use different ABCD elements, but still it’s good to get an idea: [[SampleABCDDocument]]. eb954115d045f81f5d4f9c3dc5faae5f2b4dadd6 363 362 2011-08-18T13:41:42Z JoergHoletschek 3 /* Adding Mandatory ABCD Elements */ wikitext text/x-wiki This tutorial will explain how to set up a mapping for the ABCD2.06 data schema. If you intend to map another schema (e.g. one of the ABCD extension ABCD-EFG, ABCD-DNA or HISPID), you still should take the time to read it. Large portions of the process will be identical. == Access to Biological Collection Data Schema (ABCD) == ABCD in its current version 2.06 is the data schema typically used in conjunction with BioCASe. It is a highly complex XML data schema with about 1,000 elements that is able to store almost every piece of information that can be found in a natural history specimen collection or an observation database. The complexity is due to the fact that it can be used for a wide range of collections/databases – for living and preserved specimens, observations and culture collections, for zoological, botanical, bacterial and viral collections, marine or terrestrial, for herbaria, botanic and zoological gardens. For each of these special types, ABCD features special sections where information specific for this type can be stored. Other sections of ABCD will be shared by all these types, for example for gathering site, identifier or metadata. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a list of commonly used ABCD elements on the Wiki page [[CommonABCD2Concepts|Common ABCD2 Concepts]]. == Creating an ABCD Mapping for a Data Source == If you created a data source from a template (e.g. ''abcdmetadata''), a schema mapping for ABCD will be already existent. In this case, you can directly jump to the next section. If you created an empty data source, the section <small>Schemas</small> in the data source configuration overview will be empty: [[File:abcdSchemaMappings.png]] In order to create a new schema mapping, select the desired schema from the list and click <small>Create</small>. For ABCD, choose <small>ABCD_2.06.xml</small> (be careful not to hit one of the ABCD extensions, they look quite similar). After clicking <small>Create</small>, you will be directed to the mapping editor, which will allow you to edit the mappings. == The Mapping Editor == The mapping editor allows you to add, remove and edit ABCD mappings. On top you’ll find some summary information, on the bottom you’ll see the structure of the ABCD mapping: [[File:abcdMappingEditorEmpty.png]] First thing we will do is to set the root table, which is the table that holds the records you want to publish (specimens or observations). That table must hold one record per object to be published, identified by a unique identifier (called ''UnitID''). This identifier doesn't need to be the primary key necessarily, as long as it is unique (if that sentence confuses you, ignore it and just keep in mind that the UnitID must be unique.) Use the drop down box labelled <small>Root table Alias</small> to select the root table of your data model. If your metadata table holds just one record (because all occurrences published share the same metadata) and if it is not linked to the root table by a foreign key, you can specify this table in the drop-down box <small>Static table Alias</small>. The Provider Software will use a natural join to connect the metadata record stored in this table to all occurrences published. If your metadata table is referenced by a foreign key in the root able, you must not declare the metadata table as a static table! So either join the metadata table using a foreign key OR declare it as a static table. When you’ve set the root table (and static table, if you use that), press <small>Save</small>. As with the DB structure setup, this will write your changes to the configuration files, closing the tab without saving will discard your changes! Also, in case you messed something up, you can use the <small>Revert</small> button to restore the state when you’ve saved for the last time. In the mapping editor, the link <small>Overview</small> at the top of the page will take you back to the datasource configuration overview. After creating a mapping, it will appear in the section <small>Schemas</small>. The entry will show the number of mapped elements (0 for now) and the schema namespace. Pressing the trash can symbol will move it to the trash can, which means that your web service does not support this schema anymore; if you did this by mistake, you can restore it by pressing the <small>Restore</small> button. <small>Purge</small> will delete the schema mapping permanently: [[File:abcdSchemaMappings2.png]] You can map several schemas for a given datasource. For example, in addition to the current ABCD2 standard, you could add the deprecated version 1.2 (even though I have no idea why you should do that) or DarwinCore. Just select the appropriate schema in the drop-down list and click <small>Create</small>. Since the namespace of a schema will be used in requests, you cannot create two mappings for the same namespace. Trying this will get you an error message. == Adding Mandatory ABCD Elements == Clicking on a schema in the datasource overview page will take you again to the mapping editor. In the lower half of the page, you will see the basic structure of an ABCD document: [[File:abcdMappingEditorEmptyBottom.png|border]] <strong>Technical/Content Contact:</strong> These trees hold information on who should be contacted for technical issues or questions concerning the published database, each with name, address, email address and telephone number. <strong>Metadata:</strong> In this sub tree you’ll find all metadata fields, for example title and description of the dataset, ownership information, date last modified, IPR statements etc. <strong>Units:</strong> This part stores the "real" data you want to publish, that means all specimen or observation data. The three elements already shown in the tree are identifiers that make up the so called "Triple ID", which uniquely identifies records within primary biodiversity networks such as BioCASe and GBIF: * ''UnitID'', which is a unique identifier for each record to be published (you can use the catalogue number, for example, or an existing barcode); * ''SourceInstitutionID'', an identifier for the institution publishing the dataset (for example BGBM); * ''SourceID'', an identifier for the dataset (e.g. “Herbarium Berolinense’’). The elements already shown in the mapping editor are printed in red because they are mandatory, which means they are required for an ABCD document. Without them being mapped (to database fields that are non-empty), the Provider Software will not be able to construct valid ABCD documents and will not publish any records. So the first elements that should be mapped are the mandatory fields. To add a mapping, click the <strong>+</strong> symbol next to the field, which will open the mapping editor dialog, where you can select the database table/column you want to map the ABCD element to: [[File:abcdMappingEditorDialog.png|border]] In this case, the element ''/DataSets/DataSet/Metadata/RevisionData/DateModified'' is mapped to the column ''source_update'' of the ''metadata'' table, which has the data type ''date''. (Remember that table/column retrieval is not supported for all DBMS, so if you’re unlucky, you might need to type in table/column name manually.) <small>Save</small> will accept your settings and close the dialog, <small>Cancel</small> will discard any changes. Instead of mapping an element to a database field, you can map it to a literal. For example, you could enter the technical and content contact information directly as literals in the mapping editor dialog (assuming they’re the same for all records). However, we strongly discourage you from doing this. For one thing, this is just bad style, since it will scatter the information published over different places: Some will be retrieved from the database, some (the literals) will be loaded from the configuration files. Changing the metadata of your service later would then require access to the database and access to the BioCASe configuration – that means the configuration tool password. Be nice to the person who will do these changes and put everything neatly in a database table! The second reason is even more important: Using literals for an element will result in this element to become not searchable, which means it cannot be used in a filter of a Search or a Scan request. Even though this behaviour is discoverable with a Capabilities request (by evaluating the ''searchable'' flag), a potential user of your web service might just not think of that, so it’s a good idea to avoid that in first place. Please use literals sparsely! In the mapping editor window, you can select several source columns and several literals, even a combination of both. The Provider Software will simply concatenate the values retrieved from the database and the literals specified (in the order they’re given). But again: Keep in mind that this will make the element not searchable. A better solution is to do any concatenation operations in a database view. Use the mapping editor window to add mappings for all eight mandatory fields: Click on the <strong>+</strong> symbol next to the element, choose table/column name and the correct data type, then press <small>Save</small>. When you’re done, the page should look similar to the following image. Press <small>Save</small> in the main window to write your changes to the configuration files. [[File:abcdMappingEditorMandatoryElements.png|border]] == Initial Testing of the Data Source == Once you’ve mapped all mandatory elements, you can do the first test of your BioCASe web service. Even though you haven’t mapped any real information yet, it is a good idea to check the service and solve any problems hitherto before continuing with the mapping. Click on <small>Test mapping!</small> to open the Query Form that can be used for this purpose: [[File:abcdQueryFormEmpty.png]] With the Query Form you can send a BioCASe request to your web service; it will wait for the reply of the web service and display the response document. On top of the page you can see the URL of your web service. In this case it is ''<nowiki>http://localhost/biocase/pywrapper.cgi?dsa=flora</nowiki>'', ''flora'' being the name of the datasource. Make sure this is correct for your installation! Into the text box below you can enter the request XML text. Because it would be cumbersome to enter the whole BioCASe request, you can use the links below the box to load templates. For now, click on <small>ABCD2 Search</small> to fill the box with a BioCASe Search request for documents in the ABCD2 format (the one you’ve just mapped): <syntaxhighlight> <?xml version='1.0' encoding='UTF-8'?> <request xmlns='http://www.biocase.org/schemas/protocol/1.3'> <header><type>search</type></header> <search> <requestFormat>http://www.tdwg.org/schemas/abcd/2.06</requestFormat> <responseFormat start='0' limit='10'>http://www.tdwg.org/schemas/abcd/2.06</responseFormat> <filter> <like path='/DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString'>A*</like> </filter> <count>false</count> </search> </request> </syntaxhighlight> As you can see, there is a filter set on the ''ScientificName'' element. Since we haven’t mapped this element yet, this won’t work. So let’s remove the filter (all lines between <filter> and </filter>, including the ''filter'' tags), leaving the text box like this: [[File:abcdQueryFormFilled.png]] Pressing <small>Submit</small> will send the request to the web service and – hopefully – display the response document with the correct ABCD records. For the mapping we’ve just created, it should look similar to the following (use the "–" symbol next to the trees TechnicalContact, ContentContact and Metadata to collapse them): [[File:abcdResponseMandatoryElements.png]] If you get an error, please check out the [[Debugging]] tutorial on how to debug a web service. You should continue the mapping process only after your web service returns results for the basal mapping we’ve just created. == Adding additional ABCD Elements == Once you’ve successfully managed to map the mandatory ABCD elements ("successfully" means the web service returns ABCD documents), you can go on with the mapping process. This is usually an iterative process: You add a handful of elements (for example, country, country code, locality text and coordinates for the gathering site), then you use the Query Form to see if the information ends up where it is supposed to do. This is the reason the Query Form opens up in a separate tab (or window, of you configured your web browser so): In one tab, you use the mapping editor to add new elements; after pressing <small>Save</small> you go to the Query Form tab and see what the response documents look like. Redo this until you’ve mapped all information you want to publish. In order to map new ABCD concepts, tick the check box labelled <small>Show all concepts</small> and press <small>Refresh</small>. The mapping editor will then display the whole ABCD tree, which looks admittedly a bit intimidating. In order to find the concept you’re looking for, use the search function of your browser. If you want to map the country name of the gathering site, for example, hit <small>Ctrl + F</small> (for Firefox, Microsoft IE and Opera) and type in the name of the ABCD element, namely ''country''. The browser will jump to the ''Country'' tree of the ''Gathering'' element, where you’ll find the elements for country name (''Name''), ISO country code (''ISO3166Code'') and the country name in the country’s language (''NameDerived''): [[File:abcdMappingEditorSearch.png|500px]] Continue the sequence mapping new elements/testing until you’ve mapped all information you want to publish. You can have a look at the list of recommend and commonly used ABCD elements on the [[CommonABCD2Concepts|Common ABCD2 Concepts]] page. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki]. The following page will show you a sample response ABCD document for a web service with 77 concepts mapped. Of course your documents will look different, because you will probably use different ABCD elements, but still it’s good to get an idea: [[SampleABCDDocument]]. f2fa3fecbd8a2a9fc278dee64ec4a79c7db263eb 364 363 2011-08-18T13:42:14Z JoergHoletschek 3 /* The Mapping Editor */ wikitext text/x-wiki This tutorial will explain how to set up a mapping for the ABCD2.06 data schema. If you intend to map another schema (e.g. one of the ABCD extension ABCD-EFG, ABCD-DNA or HISPID), you still should take the time to read it. Large portions of the process will be identical. == Access to Biological Collection Data Schema (ABCD) == ABCD in its current version 2.06 is the data schema typically used in conjunction with BioCASe. It is a highly complex XML data schema with about 1,000 elements that is able to store almost every piece of information that can be found in a natural history specimen collection or an observation database. The complexity is due to the fact that it can be used for a wide range of collections/databases – for living and preserved specimens, observations and culture collections, for zoological, botanical, bacterial and viral collections, marine or terrestrial, for herbaria, botanic and zoological gardens. For each of these special types, ABCD features special sections where information specific for this type can be stored. Other sections of ABCD will be shared by all these types, for example for gathering site, identifier or metadata. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a list of commonly used ABCD elements on the Wiki page [[CommonABCD2Concepts|Common ABCD2 Concepts]]. == Creating an ABCD Mapping for a Data Source == If you created a data source from a template (e.g. ''abcdmetadata''), a schema mapping for ABCD will be already existent. In this case, you can directly jump to the next section. If you created an empty data source, the section <small>Schemas</small> in the data source configuration overview will be empty: [[File:abcdSchemaMappings.png]] In order to create a new schema mapping, select the desired schema from the list and click <small>Create</small>. For ABCD, choose <small>ABCD_2.06.xml</small> (be careful not to hit one of the ABCD extensions, they look quite similar). After clicking <small>Create</small>, you will be directed to the mapping editor, which will allow you to edit the mappings. == The Mapping Editor == The mapping editor allows you to add, remove and edit ABCD mappings. On top you’ll find some summary information, on the bottom you’ll see the structure of the ABCD mapping: [[File:abcdMappingEditorEmpty.png|border]] First thing we will do is to set the root table, which is the table that holds the records you want to publish (specimens or observations). That table must hold one record per object to be published, identified by a unique identifier (called ''UnitID''). This identifier doesn't need to be the primary key necessarily, as long as it is unique (if that sentence confuses you, ignore it and just keep in mind that the UnitID must be unique.) Use the drop down box labelled <small>Root table Alias</small> to select the root table of your data model. If your metadata table holds just one record (because all occurrences published share the same metadata) and if it is not linked to the root table by a foreign key, you can specify this table in the drop-down box <small>Static table Alias</small>. The Provider Software will use a natural join to connect the metadata record stored in this table to all occurrences published. If your metadata table is referenced by a foreign key in the root able, you must not declare the metadata table as a static table! So either join the metadata table using a foreign key OR declare it as a static table. When you’ve set the root table (and static table, if you use that), press <small>Save</small>. As with the DB structure setup, this will write your changes to the configuration files, closing the tab without saving will discard your changes! Also, in case you messed something up, you can use the <small>Revert</small> button to restore the state when you’ve saved for the last time. In the mapping editor, the link <small>Overview</small> at the top of the page will take you back to the datasource configuration overview. After creating a mapping, it will appear in the section <small>Schemas</small>. The entry will show the number of mapped elements (0 for now) and the schema namespace. Pressing the trash can symbol will move it to the trash can, which means that your web service does not support this schema anymore; if you did this by mistake, you can restore it by pressing the <small>Restore</small> button. <small>Purge</small> will delete the schema mapping permanently: [[File:abcdSchemaMappings2.png|border]] You can map several schemas for a given datasource. For example, in addition to the current ABCD2 standard, you could add the deprecated version 1.2 (even though I have no idea why you should do that) or DarwinCore. Just select the appropriate schema in the drop-down list and click <small>Create</small>. Since the namespace of a schema will be used in requests, you cannot create two mappings for the same namespace. Trying this will get you an error message. == Adding Mandatory ABCD Elements == Clicking on a schema in the datasource overview page will take you again to the mapping editor. In the lower half of the page, you will see the basic structure of an ABCD document: [[File:abcdMappingEditorEmptyBottom.png|border]] <strong>Technical/Content Contact:</strong> These trees hold information on who should be contacted for technical issues or questions concerning the published database, each with name, address, email address and telephone number. <strong>Metadata:</strong> In this sub tree you’ll find all metadata fields, for example title and description of the dataset, ownership information, date last modified, IPR statements etc. <strong>Units:</strong> This part stores the "real" data you want to publish, that means all specimen or observation data. The three elements already shown in the tree are identifiers that make up the so called "Triple ID", which uniquely identifies records within primary biodiversity networks such as BioCASe and GBIF: * ''UnitID'', which is a unique identifier for each record to be published (you can use the catalogue number, for example, or an existing barcode); * ''SourceInstitutionID'', an identifier for the institution publishing the dataset (for example BGBM); * ''SourceID'', an identifier for the dataset (e.g. “Herbarium Berolinense’’). The elements already shown in the mapping editor are printed in red because they are mandatory, which means they are required for an ABCD document. Without them being mapped (to database fields that are non-empty), the Provider Software will not be able to construct valid ABCD documents and will not publish any records. So the first elements that should be mapped are the mandatory fields. To add a mapping, click the <strong>+</strong> symbol next to the field, which will open the mapping editor dialog, where you can select the database table/column you want to map the ABCD element to: [[File:abcdMappingEditorDialog.png|border]] In this case, the element ''/DataSets/DataSet/Metadata/RevisionData/DateModified'' is mapped to the column ''source_update'' of the ''metadata'' table, which has the data type ''date''. (Remember that table/column retrieval is not supported for all DBMS, so if you’re unlucky, you might need to type in table/column name manually.) <small>Save</small> will accept your settings and close the dialog, <small>Cancel</small> will discard any changes. Instead of mapping an element to a database field, you can map it to a literal. For example, you could enter the technical and content contact information directly as literals in the mapping editor dialog (assuming they’re the same for all records). However, we strongly discourage you from doing this. For one thing, this is just bad style, since it will scatter the information published over different places: Some will be retrieved from the database, some (the literals) will be loaded from the configuration files. Changing the metadata of your service later would then require access to the database and access to the BioCASe configuration – that means the configuration tool password. Be nice to the person who will do these changes and put everything neatly in a database table! The second reason is even more important: Using literals for an element will result in this element to become not searchable, which means it cannot be used in a filter of a Search or a Scan request. Even though this behaviour is discoverable with a Capabilities request (by evaluating the ''searchable'' flag), a potential user of your web service might just not think of that, so it’s a good idea to avoid that in first place. Please use literals sparsely! In the mapping editor window, you can select several source columns and several literals, even a combination of both. The Provider Software will simply concatenate the values retrieved from the database and the literals specified (in the order they’re given). But again: Keep in mind that this will make the element not searchable. A better solution is to do any concatenation operations in a database view. Use the mapping editor window to add mappings for all eight mandatory fields: Click on the <strong>+</strong> symbol next to the element, choose table/column name and the correct data type, then press <small>Save</small>. When you’re done, the page should look similar to the following image. Press <small>Save</small> in the main window to write your changes to the configuration files. [[File:abcdMappingEditorMandatoryElements.png|border]] == Initial Testing of the Data Source == Once you’ve mapped all mandatory elements, you can do the first test of your BioCASe web service. Even though you haven’t mapped any real information yet, it is a good idea to check the service and solve any problems hitherto before continuing with the mapping. Click on <small>Test mapping!</small> to open the Query Form that can be used for this purpose: [[File:abcdQueryFormEmpty.png]] With the Query Form you can send a BioCASe request to your web service; it will wait for the reply of the web service and display the response document. On top of the page you can see the URL of your web service. In this case it is ''<nowiki>http://localhost/biocase/pywrapper.cgi?dsa=flora</nowiki>'', ''flora'' being the name of the datasource. Make sure this is correct for your installation! Into the text box below you can enter the request XML text. Because it would be cumbersome to enter the whole BioCASe request, you can use the links below the box to load templates. For now, click on <small>ABCD2 Search</small> to fill the box with a BioCASe Search request for documents in the ABCD2 format (the one you’ve just mapped): <syntaxhighlight> <?xml version='1.0' encoding='UTF-8'?> <request xmlns='http://www.biocase.org/schemas/protocol/1.3'> <header><type>search</type></header> <search> <requestFormat>http://www.tdwg.org/schemas/abcd/2.06</requestFormat> <responseFormat start='0' limit='10'>http://www.tdwg.org/schemas/abcd/2.06</responseFormat> <filter> <like path='/DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString'>A*</like> </filter> <count>false</count> </search> </request> </syntaxhighlight> As you can see, there is a filter set on the ''ScientificName'' element. Since we haven’t mapped this element yet, this won’t work. So let’s remove the filter (all lines between <filter> and </filter>, including the ''filter'' tags), leaving the text box like this: [[File:abcdQueryFormFilled.png]] Pressing <small>Submit</small> will send the request to the web service and – hopefully – display the response document with the correct ABCD records. For the mapping we’ve just created, it should look similar to the following (use the "–" symbol next to the trees TechnicalContact, ContentContact and Metadata to collapse them): [[File:abcdResponseMandatoryElements.png]] If you get an error, please check out the [[Debugging]] tutorial on how to debug a web service. You should continue the mapping process only after your web service returns results for the basal mapping we’ve just created. == Adding additional ABCD Elements == Once you’ve successfully managed to map the mandatory ABCD elements ("successfully" means the web service returns ABCD documents), you can go on with the mapping process. This is usually an iterative process: You add a handful of elements (for example, country, country code, locality text and coordinates for the gathering site), then you use the Query Form to see if the information ends up where it is supposed to do. This is the reason the Query Form opens up in a separate tab (or window, of you configured your web browser so): In one tab, you use the mapping editor to add new elements; after pressing <small>Save</small> you go to the Query Form tab and see what the response documents look like. Redo this until you’ve mapped all information you want to publish. In order to map new ABCD concepts, tick the check box labelled <small>Show all concepts</small> and press <small>Refresh</small>. The mapping editor will then display the whole ABCD tree, which looks admittedly a bit intimidating. In order to find the concept you’re looking for, use the search function of your browser. If you want to map the country name of the gathering site, for example, hit <small>Ctrl + F</small> (for Firefox, Microsoft IE and Opera) and type in the name of the ABCD element, namely ''country''. The browser will jump to the ''Country'' tree of the ''Gathering'' element, where you’ll find the elements for country name (''Name''), ISO country code (''ISO3166Code'') and the country name in the country’s language (''NameDerived''): [[File:abcdMappingEditorSearch.png|500px]] Continue the sequence mapping new elements/testing until you’ve mapped all information you want to publish. You can have a look at the list of recommend and commonly used ABCD elements on the [[CommonABCD2Concepts|Common ABCD2 Concepts]] page. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki]. The following page will show you a sample response ABCD document for a web service with 77 concepts mapped. Of course your documents will look different, because you will probably use different ABCD elements, but still it’s good to get an idea: [[SampleABCDDocument]]. 361ebc51457ab5cb83fae231eef5ae6bbd33ebc6 365 364 2011-08-18T13:42:32Z JoergHoletschek 3 /* Creating an ABCD Mapping for a Data Source */ wikitext text/x-wiki This tutorial will explain how to set up a mapping for the ABCD2.06 data schema. If you intend to map another schema (e.g. one of the ABCD extension ABCD-EFG, ABCD-DNA or HISPID), you still should take the time to read it. Large portions of the process will be identical. == Access to Biological Collection Data Schema (ABCD) == ABCD in its current version 2.06 is the data schema typically used in conjunction with BioCASe. It is a highly complex XML data schema with about 1,000 elements that is able to store almost every piece of information that can be found in a natural history specimen collection or an observation database. The complexity is due to the fact that it can be used for a wide range of collections/databases – for living and preserved specimens, observations and culture collections, for zoological, botanical, bacterial and viral collections, marine or terrestrial, for herbaria, botanic and zoological gardens. For each of these special types, ABCD features special sections where information specific for this type can be stored. Other sections of ABCD will be shared by all these types, for example for gathering site, identifier or metadata. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a list of commonly used ABCD elements on the Wiki page [[CommonABCD2Concepts|Common ABCD2 Concepts]]. == Creating an ABCD Mapping for a Data Source == If you created a data source from a template (e.g. ''abcdmetadata''), a schema mapping for ABCD will be already existent. In this case, you can directly jump to the next section. If you created an empty data source, the section <small>Schemas</small> in the data source configuration overview will be empty: [[File:abcdSchemaMappings.png|border]] In order to create a new schema mapping, select the desired schema from the list and click <small>Create</small>. For ABCD, choose <small>ABCD_2.06.xml</small> (be careful not to hit one of the ABCD extensions, they look quite similar). After clicking <small>Create</small>, you will be directed to the mapping editor, which will allow you to edit the mappings. == The Mapping Editor == The mapping editor allows you to add, remove and edit ABCD mappings. On top you’ll find some summary information, on the bottom you’ll see the structure of the ABCD mapping: [[File:abcdMappingEditorEmpty.png|border]] First thing we will do is to set the root table, which is the table that holds the records you want to publish (specimens or observations). That table must hold one record per object to be published, identified by a unique identifier (called ''UnitID''). This identifier doesn't need to be the primary key necessarily, as long as it is unique (if that sentence confuses you, ignore it and just keep in mind that the UnitID must be unique.) Use the drop down box labelled <small>Root table Alias</small> to select the root table of your data model. If your metadata table holds just one record (because all occurrences published share the same metadata) and if it is not linked to the root table by a foreign key, you can specify this table in the drop-down box <small>Static table Alias</small>. The Provider Software will use a natural join to connect the metadata record stored in this table to all occurrences published. If your metadata table is referenced by a foreign key in the root able, you must not declare the metadata table as a static table! So either join the metadata table using a foreign key OR declare it as a static table. When you’ve set the root table (and static table, if you use that), press <small>Save</small>. As with the DB structure setup, this will write your changes to the configuration files, closing the tab without saving will discard your changes! Also, in case you messed something up, you can use the <small>Revert</small> button to restore the state when you’ve saved for the last time. In the mapping editor, the link <small>Overview</small> at the top of the page will take you back to the datasource configuration overview. After creating a mapping, it will appear in the section <small>Schemas</small>. The entry will show the number of mapped elements (0 for now) and the schema namespace. Pressing the trash can symbol will move it to the trash can, which means that your web service does not support this schema anymore; if you did this by mistake, you can restore it by pressing the <small>Restore</small> button. <small>Purge</small> will delete the schema mapping permanently: [[File:abcdSchemaMappings2.png|border]] You can map several schemas for a given datasource. For example, in addition to the current ABCD2 standard, you could add the deprecated version 1.2 (even though I have no idea why you should do that) or DarwinCore. Just select the appropriate schema in the drop-down list and click <small>Create</small>. Since the namespace of a schema will be used in requests, you cannot create two mappings for the same namespace. Trying this will get you an error message. == Adding Mandatory ABCD Elements == Clicking on a schema in the datasource overview page will take you again to the mapping editor. In the lower half of the page, you will see the basic structure of an ABCD document: [[File:abcdMappingEditorEmptyBottom.png|border]] <strong>Technical/Content Contact:</strong> These trees hold information on who should be contacted for technical issues or questions concerning the published database, each with name, address, email address and telephone number. <strong>Metadata:</strong> In this sub tree you’ll find all metadata fields, for example title and description of the dataset, ownership information, date last modified, IPR statements etc. <strong>Units:</strong> This part stores the "real" data you want to publish, that means all specimen or observation data. The three elements already shown in the tree are identifiers that make up the so called "Triple ID", which uniquely identifies records within primary biodiversity networks such as BioCASe and GBIF: * ''UnitID'', which is a unique identifier for each record to be published (you can use the catalogue number, for example, or an existing barcode); * ''SourceInstitutionID'', an identifier for the institution publishing the dataset (for example BGBM); * ''SourceID'', an identifier for the dataset (e.g. “Herbarium Berolinense’’). The elements already shown in the mapping editor are printed in red because they are mandatory, which means they are required for an ABCD document. Without them being mapped (to database fields that are non-empty), the Provider Software will not be able to construct valid ABCD documents and will not publish any records. So the first elements that should be mapped are the mandatory fields. To add a mapping, click the <strong>+</strong> symbol next to the field, which will open the mapping editor dialog, where you can select the database table/column you want to map the ABCD element to: [[File:abcdMappingEditorDialog.png|border]] In this case, the element ''/DataSets/DataSet/Metadata/RevisionData/DateModified'' is mapped to the column ''source_update'' of the ''metadata'' table, which has the data type ''date''. (Remember that table/column retrieval is not supported for all DBMS, so if you’re unlucky, you might need to type in table/column name manually.) <small>Save</small> will accept your settings and close the dialog, <small>Cancel</small> will discard any changes. Instead of mapping an element to a database field, you can map it to a literal. For example, you could enter the technical and content contact information directly as literals in the mapping editor dialog (assuming they’re the same for all records). However, we strongly discourage you from doing this. For one thing, this is just bad style, since it will scatter the information published over different places: Some will be retrieved from the database, some (the literals) will be loaded from the configuration files. Changing the metadata of your service later would then require access to the database and access to the BioCASe configuration – that means the configuration tool password. Be nice to the person who will do these changes and put everything neatly in a database table! The second reason is even more important: Using literals for an element will result in this element to become not searchable, which means it cannot be used in a filter of a Search or a Scan request. Even though this behaviour is discoverable with a Capabilities request (by evaluating the ''searchable'' flag), a potential user of your web service might just not think of that, so it’s a good idea to avoid that in first place. Please use literals sparsely! In the mapping editor window, you can select several source columns and several literals, even a combination of both. The Provider Software will simply concatenate the values retrieved from the database and the literals specified (in the order they’re given). But again: Keep in mind that this will make the element not searchable. A better solution is to do any concatenation operations in a database view. Use the mapping editor window to add mappings for all eight mandatory fields: Click on the <strong>+</strong> symbol next to the element, choose table/column name and the correct data type, then press <small>Save</small>. When you’re done, the page should look similar to the following image. Press <small>Save</small> in the main window to write your changes to the configuration files. [[File:abcdMappingEditorMandatoryElements.png|border]] == Initial Testing of the Data Source == Once you’ve mapped all mandatory elements, you can do the first test of your BioCASe web service. Even though you haven’t mapped any real information yet, it is a good idea to check the service and solve any problems hitherto before continuing with the mapping. Click on <small>Test mapping!</small> to open the Query Form that can be used for this purpose: [[File:abcdQueryFormEmpty.png]] With the Query Form you can send a BioCASe request to your web service; it will wait for the reply of the web service and display the response document. On top of the page you can see the URL of your web service. In this case it is ''<nowiki>http://localhost/biocase/pywrapper.cgi?dsa=flora</nowiki>'', ''flora'' being the name of the datasource. Make sure this is correct for your installation! Into the text box below you can enter the request XML text. Because it would be cumbersome to enter the whole BioCASe request, you can use the links below the box to load templates. For now, click on <small>ABCD2 Search</small> to fill the box with a BioCASe Search request for documents in the ABCD2 format (the one you’ve just mapped): <syntaxhighlight> <?xml version='1.0' encoding='UTF-8'?> <request xmlns='http://www.biocase.org/schemas/protocol/1.3'> <header><type>search</type></header> <search> <requestFormat>http://www.tdwg.org/schemas/abcd/2.06</requestFormat> <responseFormat start='0' limit='10'>http://www.tdwg.org/schemas/abcd/2.06</responseFormat> <filter> <like path='/DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString'>A*</like> </filter> <count>false</count> </search> </request> </syntaxhighlight> As you can see, there is a filter set on the ''ScientificName'' element. Since we haven’t mapped this element yet, this won’t work. So let’s remove the filter (all lines between <filter> and </filter>, including the ''filter'' tags), leaving the text box like this: [[File:abcdQueryFormFilled.png]] Pressing <small>Submit</small> will send the request to the web service and – hopefully – display the response document with the correct ABCD records. For the mapping we’ve just created, it should look similar to the following (use the "–" symbol next to the trees TechnicalContact, ContentContact and Metadata to collapse them): [[File:abcdResponseMandatoryElements.png]] If you get an error, please check out the [[Debugging]] tutorial on how to debug a web service. You should continue the mapping process only after your web service returns results for the basal mapping we’ve just created. == Adding additional ABCD Elements == Once you’ve successfully managed to map the mandatory ABCD elements ("successfully" means the web service returns ABCD documents), you can go on with the mapping process. This is usually an iterative process: You add a handful of elements (for example, country, country code, locality text and coordinates for the gathering site), then you use the Query Form to see if the information ends up where it is supposed to do. This is the reason the Query Form opens up in a separate tab (or window, of you configured your web browser so): In one tab, you use the mapping editor to add new elements; after pressing <small>Save</small> you go to the Query Form tab and see what the response documents look like. Redo this until you’ve mapped all information you want to publish. In order to map new ABCD concepts, tick the check box labelled <small>Show all concepts</small> and press <small>Refresh</small>. The mapping editor will then display the whole ABCD tree, which looks admittedly a bit intimidating. In order to find the concept you’re looking for, use the search function of your browser. If you want to map the country name of the gathering site, for example, hit <small>Ctrl + F</small> (for Firefox, Microsoft IE and Opera) and type in the name of the ABCD element, namely ''country''. The browser will jump to the ''Country'' tree of the ''Gathering'' element, where you’ll find the elements for country name (''Name''), ISO country code (''ISO3166Code'') and the country name in the country’s language (''NameDerived''): [[File:abcdMappingEditorSearch.png|500px]] Continue the sequence mapping new elements/testing until you’ve mapped all information you want to publish. You can have a look at the list of recommend and commonly used ABCD elements on the [[CommonABCD2Concepts|Common ABCD2 Concepts]] page. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki]. The following page will show you a sample response ABCD document for a web service with 77 concepts mapped. Of course your documents will look different, because you will probably use different ABCD elements, but still it’s good to get an idea: [[SampleABCDDocument]]. 60032ac8833e57238c56e8ff89fdff1a6f562e25 366 365 2011-08-18T13:43:12Z JoergHoletschek 3 /* Initial Testing of the Data Source */ wikitext text/x-wiki This tutorial will explain how to set up a mapping for the ABCD2.06 data schema. If you intend to map another schema (e.g. one of the ABCD extension ABCD-EFG, ABCD-DNA or HISPID), you still should take the time to read it. Large portions of the process will be identical. == Access to Biological Collection Data Schema (ABCD) == ABCD in its current version 2.06 is the data schema typically used in conjunction with BioCASe. It is a highly complex XML data schema with about 1,000 elements that is able to store almost every piece of information that can be found in a natural history specimen collection or an observation database. The complexity is due to the fact that it can be used for a wide range of collections/databases – for living and preserved specimens, observations and culture collections, for zoological, botanical, bacterial and viral collections, marine or terrestrial, for herbaria, botanic and zoological gardens. For each of these special types, ABCD features special sections where information specific for this type can be stored. Other sections of ABCD will be shared by all these types, for example for gathering site, identifier or metadata. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a list of commonly used ABCD elements on the Wiki page [[CommonABCD2Concepts|Common ABCD2 Concepts]]. == Creating an ABCD Mapping for a Data Source == If you created a data source from a template (e.g. ''abcdmetadata''), a schema mapping for ABCD will be already existent. In this case, you can directly jump to the next section. If you created an empty data source, the section <small>Schemas</small> in the data source configuration overview will be empty: [[File:abcdSchemaMappings.png|border]] In order to create a new schema mapping, select the desired schema from the list and click <small>Create</small>. For ABCD, choose <small>ABCD_2.06.xml</small> (be careful not to hit one of the ABCD extensions, they look quite similar). After clicking <small>Create</small>, you will be directed to the mapping editor, which will allow you to edit the mappings. == The Mapping Editor == The mapping editor allows you to add, remove and edit ABCD mappings. On top you’ll find some summary information, on the bottom you’ll see the structure of the ABCD mapping: [[File:abcdMappingEditorEmpty.png|border]] First thing we will do is to set the root table, which is the table that holds the records you want to publish (specimens or observations). That table must hold one record per object to be published, identified by a unique identifier (called ''UnitID''). This identifier doesn't need to be the primary key necessarily, as long as it is unique (if that sentence confuses you, ignore it and just keep in mind that the UnitID must be unique.) Use the drop down box labelled <small>Root table Alias</small> to select the root table of your data model. If your metadata table holds just one record (because all occurrences published share the same metadata) and if it is not linked to the root table by a foreign key, you can specify this table in the drop-down box <small>Static table Alias</small>. The Provider Software will use a natural join to connect the metadata record stored in this table to all occurrences published. If your metadata table is referenced by a foreign key in the root able, you must not declare the metadata table as a static table! So either join the metadata table using a foreign key OR declare it as a static table. When you’ve set the root table (and static table, if you use that), press <small>Save</small>. As with the DB structure setup, this will write your changes to the configuration files, closing the tab without saving will discard your changes! Also, in case you messed something up, you can use the <small>Revert</small> button to restore the state when you’ve saved for the last time. In the mapping editor, the link <small>Overview</small> at the top of the page will take you back to the datasource configuration overview. After creating a mapping, it will appear in the section <small>Schemas</small>. The entry will show the number of mapped elements (0 for now) and the schema namespace. Pressing the trash can symbol will move it to the trash can, which means that your web service does not support this schema anymore; if you did this by mistake, you can restore it by pressing the <small>Restore</small> button. <small>Purge</small> will delete the schema mapping permanently: [[File:abcdSchemaMappings2.png|border]] You can map several schemas for a given datasource. For example, in addition to the current ABCD2 standard, you could add the deprecated version 1.2 (even though I have no idea why you should do that) or DarwinCore. Just select the appropriate schema in the drop-down list and click <small>Create</small>. Since the namespace of a schema will be used in requests, you cannot create two mappings for the same namespace. Trying this will get you an error message. == Adding Mandatory ABCD Elements == Clicking on a schema in the datasource overview page will take you again to the mapping editor. In the lower half of the page, you will see the basic structure of an ABCD document: [[File:abcdMappingEditorEmptyBottom.png|border]] <strong>Technical/Content Contact:</strong> These trees hold information on who should be contacted for technical issues or questions concerning the published database, each with name, address, email address and telephone number. <strong>Metadata:</strong> In this sub tree you’ll find all metadata fields, for example title and description of the dataset, ownership information, date last modified, IPR statements etc. <strong>Units:</strong> This part stores the "real" data you want to publish, that means all specimen or observation data. The three elements already shown in the tree are identifiers that make up the so called "Triple ID", which uniquely identifies records within primary biodiversity networks such as BioCASe and GBIF: * ''UnitID'', which is a unique identifier for each record to be published (you can use the catalogue number, for example, or an existing barcode); * ''SourceInstitutionID'', an identifier for the institution publishing the dataset (for example BGBM); * ''SourceID'', an identifier for the dataset (e.g. “Herbarium Berolinense’’). The elements already shown in the mapping editor are printed in red because they are mandatory, which means they are required for an ABCD document. Without them being mapped (to database fields that are non-empty), the Provider Software will not be able to construct valid ABCD documents and will not publish any records. So the first elements that should be mapped are the mandatory fields. To add a mapping, click the <strong>+</strong> symbol next to the field, which will open the mapping editor dialog, where you can select the database table/column you want to map the ABCD element to: [[File:abcdMappingEditorDialog.png|border]] In this case, the element ''/DataSets/DataSet/Metadata/RevisionData/DateModified'' is mapped to the column ''source_update'' of the ''metadata'' table, which has the data type ''date''. (Remember that table/column retrieval is not supported for all DBMS, so if you’re unlucky, you might need to type in table/column name manually.) <small>Save</small> will accept your settings and close the dialog, <small>Cancel</small> will discard any changes. Instead of mapping an element to a database field, you can map it to a literal. For example, you could enter the technical and content contact information directly as literals in the mapping editor dialog (assuming they’re the same for all records). However, we strongly discourage you from doing this. For one thing, this is just bad style, since it will scatter the information published over different places: Some will be retrieved from the database, some (the literals) will be loaded from the configuration files. Changing the metadata of your service later would then require access to the database and access to the BioCASe configuration – that means the configuration tool password. Be nice to the person who will do these changes and put everything neatly in a database table! The second reason is even more important: Using literals for an element will result in this element to become not searchable, which means it cannot be used in a filter of a Search or a Scan request. Even though this behaviour is discoverable with a Capabilities request (by evaluating the ''searchable'' flag), a potential user of your web service might just not think of that, so it’s a good idea to avoid that in first place. Please use literals sparsely! In the mapping editor window, you can select several source columns and several literals, even a combination of both. The Provider Software will simply concatenate the values retrieved from the database and the literals specified (in the order they’re given). But again: Keep in mind that this will make the element not searchable. A better solution is to do any concatenation operations in a database view. Use the mapping editor window to add mappings for all eight mandatory fields: Click on the <strong>+</strong> symbol next to the element, choose table/column name and the correct data type, then press <small>Save</small>. When you’re done, the page should look similar to the following image. Press <small>Save</small> in the main window to write your changes to the configuration files. [[File:abcdMappingEditorMandatoryElements.png|border]] == Initial Testing of the Data Source == Once you’ve mapped all mandatory elements, you can do the first test of your BioCASe web service. Even though you haven’t mapped any real information yet, it is a good idea to check the service and solve any problems hitherto before continuing with the mapping. Click on <small>Test mapping!</small> to open the Query Form that can be used for this purpose: [[File:abcdQueryFormEmpty.png|border]] With the Query Form you can send a BioCASe request to your web service; it will wait for the reply of the web service and display the response document. On top of the page you can see the URL of your web service. In this case it is ''<nowiki>http://localhost/biocase/pywrapper.cgi?dsa=flora</nowiki>'', ''flora'' being the name of the datasource. Make sure this is correct for your installation! Into the text box below you can enter the request XML text. Because it would be cumbersome to enter the whole BioCASe request, you can use the links below the box to load templates. For now, click on <small>ABCD2 Search</small> to fill the box with a BioCASe Search request for documents in the ABCD2 format (the one you’ve just mapped): <syntaxhighlight> <?xml version='1.0' encoding='UTF-8'?> <request xmlns='http://www.biocase.org/schemas/protocol/1.3'> <header><type>search</type></header> <search> <requestFormat>http://www.tdwg.org/schemas/abcd/2.06</requestFormat> <responseFormat start='0' limit='10'>http://www.tdwg.org/schemas/abcd/2.06</responseFormat> <filter> <like path='/DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString'>A*</like> </filter> <count>false</count> </search> </request> </syntaxhighlight> As you can see, there is a filter set on the ''ScientificName'' element. Since we haven’t mapped this element yet, this won’t work. So let’s remove the filter (all lines between <filter> and </filter>, including the ''filter'' tags), leaving the text box like this: [[File:abcdQueryFormFilled.png|border]] Pressing <small>Submit</small> will send the request to the web service and – hopefully – display the response document with the correct ABCD records. For the mapping we’ve just created, it should look similar to the following (use the "–" symbol next to the trees TechnicalContact, ContentContact and Metadata to collapse them): [[File:abcdResponseMandatoryElements.png|border]] If you get an error, please check out the [[Debugging]] tutorial on how to debug a web service. You should continue the mapping process only after your web service returns results for the basal mapping we’ve just created. == Adding additional ABCD Elements == Once you’ve successfully managed to map the mandatory ABCD elements ("successfully" means the web service returns ABCD documents), you can go on with the mapping process. This is usually an iterative process: You add a handful of elements (for example, country, country code, locality text and coordinates for the gathering site), then you use the Query Form to see if the information ends up where it is supposed to do. This is the reason the Query Form opens up in a separate tab (or window, of you configured your web browser so): In one tab, you use the mapping editor to add new elements; after pressing <small>Save</small> you go to the Query Form tab and see what the response documents look like. Redo this until you’ve mapped all information you want to publish. In order to map new ABCD concepts, tick the check box labelled <small>Show all concepts</small> and press <small>Refresh</small>. The mapping editor will then display the whole ABCD tree, which looks admittedly a bit intimidating. In order to find the concept you’re looking for, use the search function of your browser. If you want to map the country name of the gathering site, for example, hit <small>Ctrl + F</small> (for Firefox, Microsoft IE and Opera) and type in the name of the ABCD element, namely ''country''. The browser will jump to the ''Country'' tree of the ''Gathering'' element, where you’ll find the elements for country name (''Name''), ISO country code (''ISO3166Code'') and the country name in the country’s language (''NameDerived''): [[File:abcdMappingEditorSearch.png|500px]] Continue the sequence mapping new elements/testing until you’ve mapped all information you want to publish. You can have a look at the list of recommend and commonly used ABCD elements on the [[CommonABCD2Concepts|Common ABCD2 Concepts]] page. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki]. The following page will show you a sample response ABCD document for a web service with 77 concepts mapped. Of course your documents will look different, because you will probably use different ABCD elements, but still it’s good to get an idea: [[SampleABCDDocument]]. ee0d90e54e22b1d225a8bcf8372627907a6d281c 0 41 367 361 2011-08-18T13:43:58Z JoergHoletschek 3 /* Debugging a BioCASe Web Service */ wikitext text/x-wiki This tutorial explains how to debug a BioCASe web service and a BioCASe installation in general. If you follow the instructions and don’t manage to get your service running, please [[Special:Contact|contact the BioCASe team]]. == Debugging a BioCASe Web Service == Debugging a BioCASe web service can be done with the manual Query Form. It allows you to load a BioCASe request (Search/Scan) from a template, edit the parameters of the request (e.g. the filter) and will show you the response document, which holds the requested ABCD records and useful debug information. === Opening the Query Form === There are several ways to open the Query Form. On the entry page of a datasource, <small>TEST and DEBUG</small> will open the Query Form: [[File:debugDatasourceEntryPage.png|border]] In the data source configuration tool, the link to the according query form can be always found in the link list at the top of the page. Moreover, in the mapping editor, the button <small>Test mapping!</small> will open the Query Form in a separate tab: [[File:abcdMappingEditorEmptyTop.png|border]] === Basic Usage === On top of the Query Form you can see the URL of your web service. In this case it is ''<nowiki>http://localhost/biocase/pywrapper.cgi?dsa=flora</nowiki>'', ''flora'' being the name of the datasource. Make sure this is the correct access point of the web service you want to debug! [[File:debugQueryformFull.png|border]] The usage of the Query Form is very straightforward: * Click on one of the links below the text box to load a template, for example for an ABCD2 Scan or ABCD2 Search; * Edit the request parameters; * Click on <small>Submit</small> to get the response document. The section <small>BioCASe protocol filter operators</small> lists the operators that can be used for creating complex filters that combine several filter criteria. More about creating filters can be found in the BioCASe Protocol documentation at http://www.biocase.org/products/protocols/index.shtml. Clicking on the <small>Show</small> link in the <small>Capabilities</small> section will show the Capabilities response in a frame of the Query Form. This is useful, since it shows the namespaces for the schemas supported by the web service (which is required in the request text) and XPaths of mapped ABCD elements (required for creating filters on these elements). By using Copy & Paste you can spare the trouble of entering these monstrous strings of characters. === Changing the Debug Level for a Data Source === Debug info can be set to one of four levels for a BioCASe data source: * <strong>Error</strong> is the highest level will only output errors that prevent the web server from answering a request. * <strong>Warning</strong> will output errors that keep BioCASe from answering a request and warnings that don’t prevent the fulfilling of the request, but should be analysed carefully. For example, they might tell you that records have been dropped because mandatory ABCD elements have not been mapped or were empty for certain records. * <strong>Info</strong> is the default and will add some more information about how the Provider Software processes the request. * <strong>Debug</strong> is the most detailed level and will produce very verbose debug output. It adds very comprehensive information about how the ABCD documents are constructed and is suitable only for experienced users. For most cases, keeping the default ''Info'' level should be sufficient for testing and debugging a web service. For production use, you should set it to ''Error'' later. You can do this in the configuration of a datasource under Settings: [[File:debugDatasourceSettings.png|border]] == Analyzing the Response Document == Upon clicking <small>Submit</small>, the Query Form will send the request to the web service and wait for the response. It might take some time – up to minutes – until the response document shows up. Amongst others, this depends on * the complexity of the query (especially the filters), * the size of the dataset returned, * the debug level set (“debug” results in very exhaustive debug output), * the performance of the servers running BioCASe and the database, * the network bandwidth. The response document consists of three sections (a full response document can be found [[SampleABCDDocument|here]]): [[File:debugResponseDocumentOverview.png]] === Header === This section contains information about the BioCASe installation – operating system, DBMS used, Python/BioCASe version etc. It is intended for a consumer application and won’t be very helpful for debugging: <syntaxhighlight> <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> </syntaxhighlight> === Content === This section makes up the payload of the response document, depending on the type of request: <strong>Capabilities:</strong> Lists all schemas supported by the web service. For each schema, all mapped concepts (identified by their XPaths) are listed with data type and searchable flag. <syntaxhighlight> <biocase:content recordCount="0" recordDropped="0" recordStart="0" totalSearchHits="0"> <biocase:capabilities> <biocase:SupportedSchemas namespace="http://www.tdwg.org/schemas/abcd/2.06" request="true" response="true"> <biocase:Concept datatype="normalizedString" searchable="1">/DataSets/DataSet/ContentContacts/ContentContact/Address</biocase:Concept> <biocase:Concept datatype="normalizedString" searchable="1">/DataSets/DataSet/ContentContacts/ContentContact/Email</biocase:Concept> <biocase:Concept datatype="normalizedString" searchable="1">/DataSets/DataSet/ContentContacts/ContentContact/Name</biocase:Concept> <biocase:Concept datatype="normalizedString" searchable="1">/DataSets/DataSet/Metadata/Description/Representation/Details</biocase:Concept> ... </biocase:SupportedSchemas> </biocase:capabilities> </biocase:content> </syntaxhighlight> <strong>Scan:</strong> The document will simply list all distinct values for the concept specified in the request. When done for the scientific name, it would look similar to this: <syntaxhighlight> <biocase:content recordCount="1664" recordDropped="0" recordStart="0" totalSearchHits="1664"> <biocase:scan> <biocase:value>Achillea atrata L.</biocase:value> <biocase:value>Achillea millefolium L.</biocase:value> <biocase:value>Achillea nobilis L. ssp. nobilis</biocase:value> <biocase:value>Achillea ptarmica L.</biocase:value> <biocase:value>Acinos alpinus (L.) Moench</biocase:value> <biocase:value>Acinos arvensis (Lam.) Dandy</biocase:value> <biocase:value>Aconitum degenii ssp. paniculatum (Arcang.) Mucher</biocase:value> <biocase:value>Aconitum lycoctonum L. ssp. lycoctonum</biocase:value> <biocase:value>Aconitum napellus L. s.l.</biocase:value> ... </biocase:scan> </biocase:content> </syntaxhighlight> <strong>Search:</strong> For this type of request, the content consists of the ABCD dataset(s). See the the [[SampleABCDDocument| ABCD2 example]]. === Diagnostics (Debug Output) === This section is the most interesting part for debugging. If debug level is set to ''Info'' (which is recommended for debugging), it will document the processing of the request by the web service. The output is easy to understand, for example it will contain the SQL statements sent to the database. For a successful search request it could look similar to this: <syntaxhighlight> <biocase:diagnostics> <biocase:diagnostic severity="INFO"> Datasource wrapper FloraExsiccataBavarica requested </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Reading PSF from C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica \provider_setup_file.xml </biocase:diagnostic> <biocase:diagnostic severity="DEBUG"> PSF: PSF=C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\provider_setup_file.xml, recLimit=100, loglevel=20, user=webuser, database=rbg, dbIP=192.168.2.10, dbms=mysql, encoding=latin_1, schemas={u'http://www.tdwg.org/schemas/abcd/2.06': <biocase.wrapper.psf_handler.SupportedSchema instance at 0x011F5B98>}, tablegraph=GRAPH: graph: unit-metadata, +++ ALIAS2TABLE: {u'unit': u'feb3', u'metadata': u'metadata'} </biocase:diagnostic> <biocase:diagnostic severity="INFO"> BioCASe protocol used. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> HTTP parameter 'query' used for building the request. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT DISTINCT unit.id FROM feb3 AS unit WHERE (unit.nameautor LIKE 'Allium %' ) LIMIT 11' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT unit.id, metadata.supplier_organisation, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (40, 41, 42, 43, 44, 45, 46, 47)) ORDER BY unit.id' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> time to execute request is 0.46599984169 </biocase:diagnostic> </biocase:diagnostics> </syntaxhighlight> == Sample Errors and Warnings in the Debug Output == <syntaxhighlight> <biocase:diagnostic severity="ERROR"> No DB connection could be established. Please verify DB state and connection parameters in your provider_setup_file.xml. </biocase:diagnostic> </syntaxhighlight> Well... just what it says. The Provider Software could not connect to the database to be published. Check the database connection settings (server IP, database name, user, and password) and make sure you’ve installed the Python package for the DBMS you’re using (see the <small>Libs test</small> page in <small>Utilities</small>). <syntaxhighlight> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\flora\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="ERROR"> The CMF DB mapping file http://www.tdwg.org/schemas/abcd/2.06 was corrupt or could not be interpreted. </biocase:diagnostic> </syntaxhighlight> The requested schema (identified by the namespace http://www.tdwg.org/schemas/abcd/2.06) is not supported by that service. Create a mapping for this schema and map elements. <syntaxhighlight> <biocase:diagnostic severity="ERROR"> The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result /TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts </biocase:diagnostic> </syntaxhighlight> Obvious - happens if you try to use a concept that is not mapped in a scan request or a filter. <syntaxhighlight> <biocase:diagnostic severity="ERROR"> A SQL statement produced an error: SELECT unit.id, metadata.supplier_organisation, metadata.supplier_email, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) ORDER BY unit.id </biocase:diagnostic> </syntaxhighlight> That could have a number of reasons: One of the table/column names that were set up in the configuration does not exist (because it was renamed or removed) or the credentials used by the BPS do not have sufficient privileges. Simply copy the SQL statement and execute it manually on the database with a regular database client that will show you the detailed error message returned by the DBMS. <syntaxhighlight> <biocase:diagnostic severity="WARNING"> The mandatory child element SourceInstitutionID could not be created [0 existing, but 1 required]. The element /DataSets/DataSet/Units/Unit was also dropped. </biocase:diagnostic> <biocase:diagnostic severity="WARNING"> The mandatory child element Units could not be created [0 existing, but 1 required]. The element /DataSets/DataSet was also dropped. </biocase:diagnostic> </syntaxhighlight> One problem caused the next one in this sequence of warnings: The mandatory ABCD element ''SourceInstitutionID '' was either not mapped or empty; therefore the respective record (''unit'') was dropped. Because this happened for all units in the dataset, all units were dropped, resulting in an empty ''Units'' sub tree. The second warning tells you what happened then: Because the ''Units'' element is mandatory for the ''Dataset'' element, the whole dataset was dropped. As a result, the response ABCD document was empty. == Debugging a BioCASe Installation == In case the debug output of the web service doesn’t help you to find the problem, or if the BPS behaves not as intended without displaying any error messages, you should have a look at the debug logs written by the BPS. To turn on the Debug logs, go to the System Administration page and set <small>Debug logs</small> to <small>True</small>. [[File:instConfigTool.png|500px]] Redo the step that didn’t work and open the ''log'' folder of your BioCASe installation. There are several global log files (pywrapper_request.log, pywrapper_error.log, webapp_debug.log, webapp_error.log) and a debug files per data source (e.g. debug_flora.log). Open these files and see if they can give you a clue to the problem. If you still can’t get it running, [[Special:Contact|contact the BioCASe team]]. 5af44835c37bf978bf16a9c1082c9d61438a0c53 368 367 2011-08-18T13:44:29Z JoergHoletschek 3 /* Analyzing the Response Document */ wikitext text/x-wiki This tutorial explains how to debug a BioCASe web service and a BioCASe installation in general. If you follow the instructions and don’t manage to get your service running, please [[Special:Contact|contact the BioCASe team]]. == Debugging a BioCASe Web Service == Debugging a BioCASe web service can be done with the manual Query Form. It allows you to load a BioCASe request (Search/Scan) from a template, edit the parameters of the request (e.g. the filter) and will show you the response document, which holds the requested ABCD records and useful debug information. === Opening the Query Form === There are several ways to open the Query Form. On the entry page of a datasource, <small>TEST and DEBUG</small> will open the Query Form: [[File:debugDatasourceEntryPage.png|border]] In the data source configuration tool, the link to the according query form can be always found in the link list at the top of the page. Moreover, in the mapping editor, the button <small>Test mapping!</small> will open the Query Form in a separate tab: [[File:abcdMappingEditorEmptyTop.png|border]] === Basic Usage === On top of the Query Form you can see the URL of your web service. In this case it is ''<nowiki>http://localhost/biocase/pywrapper.cgi?dsa=flora</nowiki>'', ''flora'' being the name of the datasource. Make sure this is the correct access point of the web service you want to debug! [[File:debugQueryformFull.png|border]] The usage of the Query Form is very straightforward: * Click on one of the links below the text box to load a template, for example for an ABCD2 Scan or ABCD2 Search; * Edit the request parameters; * Click on <small>Submit</small> to get the response document. The section <small>BioCASe protocol filter operators</small> lists the operators that can be used for creating complex filters that combine several filter criteria. More about creating filters can be found in the BioCASe Protocol documentation at http://www.biocase.org/products/protocols/index.shtml. Clicking on the <small>Show</small> link in the <small>Capabilities</small> section will show the Capabilities response in a frame of the Query Form. This is useful, since it shows the namespaces for the schemas supported by the web service (which is required in the request text) and XPaths of mapped ABCD elements (required for creating filters on these elements). By using Copy & Paste you can spare the trouble of entering these monstrous strings of characters. === Changing the Debug Level for a Data Source === Debug info can be set to one of four levels for a BioCASe data source: * <strong>Error</strong> is the highest level will only output errors that prevent the web server from answering a request. * <strong>Warning</strong> will output errors that keep BioCASe from answering a request and warnings that don’t prevent the fulfilling of the request, but should be analysed carefully. For example, they might tell you that records have been dropped because mandatory ABCD elements have not been mapped or were empty for certain records. * <strong>Info</strong> is the default and will add some more information about how the Provider Software processes the request. * <strong>Debug</strong> is the most detailed level and will produce very verbose debug output. It adds very comprehensive information about how the ABCD documents are constructed and is suitable only for experienced users. For most cases, keeping the default ''Info'' level should be sufficient for testing and debugging a web service. For production use, you should set it to ''Error'' later. You can do this in the configuration of a datasource under Settings: [[File:debugDatasourceSettings.png|border]] == Analyzing the Response Document == Upon clicking <small>Submit</small>, the Query Form will send the request to the web service and wait for the response. It might take some time – up to minutes – until the response document shows up. Amongst others, this depends on * the complexity of the query (especially the filters), * the size of the dataset returned, * the debug level set (“debug” results in very exhaustive debug output), * the performance of the servers running BioCASe and the database, * the network bandwidth. The response document consists of three sections (a full response document can be found [[SampleABCDDocument|here]]): [[File:debugResponseDocumentOverview.png|border]] === Header === This section contains information about the BioCASe installation – operating system, DBMS used, Python/BioCASe version etc. It is intended for a consumer application and won’t be very helpful for debugging: <syntaxhighlight> <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> </syntaxhighlight> === Content === This section makes up the payload of the response document, depending on the type of request: <strong>Capabilities:</strong> Lists all schemas supported by the web service. For each schema, all mapped concepts (identified by their XPaths) are listed with data type and searchable flag. <syntaxhighlight> <biocase:content recordCount="0" recordDropped="0" recordStart="0" totalSearchHits="0"> <biocase:capabilities> <biocase:SupportedSchemas namespace="http://www.tdwg.org/schemas/abcd/2.06" request="true" response="true"> <biocase:Concept datatype="normalizedString" searchable="1">/DataSets/DataSet/ContentContacts/ContentContact/Address</biocase:Concept> <biocase:Concept datatype="normalizedString" searchable="1">/DataSets/DataSet/ContentContacts/ContentContact/Email</biocase:Concept> <biocase:Concept datatype="normalizedString" searchable="1">/DataSets/DataSet/ContentContacts/ContentContact/Name</biocase:Concept> <biocase:Concept datatype="normalizedString" searchable="1">/DataSets/DataSet/Metadata/Description/Representation/Details</biocase:Concept> ... </biocase:SupportedSchemas> </biocase:capabilities> </biocase:content> </syntaxhighlight> <strong>Scan:</strong> The document will simply list all distinct values for the concept specified in the request. When done for the scientific name, it would look similar to this: <syntaxhighlight> <biocase:content recordCount="1664" recordDropped="0" recordStart="0" totalSearchHits="1664"> <biocase:scan> <biocase:value>Achillea atrata L.</biocase:value> <biocase:value>Achillea millefolium L.</biocase:value> <biocase:value>Achillea nobilis L. ssp. nobilis</biocase:value> <biocase:value>Achillea ptarmica L.</biocase:value> <biocase:value>Acinos alpinus (L.) Moench</biocase:value> <biocase:value>Acinos arvensis (Lam.) Dandy</biocase:value> <biocase:value>Aconitum degenii ssp. paniculatum (Arcang.) Mucher</biocase:value> <biocase:value>Aconitum lycoctonum L. ssp. lycoctonum</biocase:value> <biocase:value>Aconitum napellus L. s.l.</biocase:value> ... </biocase:scan> </biocase:content> </syntaxhighlight> <strong>Search:</strong> For this type of request, the content consists of the ABCD dataset(s). See the the [[SampleABCDDocument| ABCD2 example]]. === Diagnostics (Debug Output) === This section is the most interesting part for debugging. If debug level is set to ''Info'' (which is recommended for debugging), it will document the processing of the request by the web service. The output is easy to understand, for example it will contain the SQL statements sent to the database. For a successful search request it could look similar to this: <syntaxhighlight> <biocase:diagnostics> <biocase:diagnostic severity="INFO"> Datasource wrapper FloraExsiccataBavarica requested </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Reading PSF from C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica \provider_setup_file.xml </biocase:diagnostic> <biocase:diagnostic severity="DEBUG"> PSF: PSF=C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\provider_setup_file.xml, recLimit=100, loglevel=20, user=webuser, database=rbg, dbIP=192.168.2.10, dbms=mysql, encoding=latin_1, schemas={u'http://www.tdwg.org/schemas/abcd/2.06': <biocase.wrapper.psf_handler.SupportedSchema instance at 0x011F5B98>}, tablegraph=GRAPH: graph: unit-metadata, +++ ALIAS2TABLE: {u'unit': u'feb3', u'metadata': u'metadata'} </biocase:diagnostic> <biocase:diagnostic severity="INFO"> BioCASe protocol used. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> HTTP parameter 'query' used for building the request. </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\FloraExsiccataBavarica\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT DISTINCT unit.id FROM feb3 AS unit WHERE (unit.nameautor LIKE 'Allium %' ) LIMIT 11' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Executing SQL: 'SELECT unit.id, metadata.supplier_organisation, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (40, 41, 42, 43, 44, 45, 46, 47)) ORDER BY unit.id' </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Hits: 8 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> time to execute request is 0.46599984169 </biocase:diagnostic> </biocase:diagnostics> </syntaxhighlight> == Sample Errors and Warnings in the Debug Output == <syntaxhighlight> <biocase:diagnostic severity="ERROR"> No DB connection could be established. Please verify DB state and connection parameters in your provider_setup_file.xml. </biocase:diagnostic> </syntaxhighlight> Well... just what it says. The Provider Software could not connect to the database to be published. Check the database connection settings (server IP, database name, user, and password) and make sure you’ve installed the Python package for the DBMS you’re using (see the <small>Libs test</small> page in <small>Utilities</small>). <syntaxhighlight> <biocase:diagnostic severity="INFO"> Try to get CMF for namespace http://www.tdwg.org/schemas/abcd/2.06 </biocase:diagnostic> <biocase:diagnostic severity="INFO"> Load CMFile 'C:\Workspace\bps2\config\datasources\flora\cmf_ABCD_2.06.xml' </biocase:diagnostic> <biocase:diagnostic severity="ERROR"> The CMF DB mapping file http://www.tdwg.org/schemas/abcd/2.06 was corrupt or could not be interpreted. </biocase:diagnostic> </syntaxhighlight> The requested schema (identified by the namespace http://www.tdwg.org/schemas/abcd/2.06) is not supported by that service. Create a mapping for this schema and map elements. <syntaxhighlight> <biocase:diagnostic severity="ERROR"> The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result /TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts </biocase:diagnostic> </syntaxhighlight> Obvious - happens if you try to use a concept that is not mapped in a scan request or a filter. <syntaxhighlight> <biocase:diagnostic severity="ERROR"> A SQL statement produced an error: SELECT unit.id, metadata.supplier_organisation, metadata.supplier_email, ... FROM feb3 AS unit LEFT JOIN metadata AS metadata ON (metadata.id = unit.metadata_fk) WHERE (unit.id IN (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) ORDER BY unit.id </biocase:diagnostic> </syntaxhighlight> That could have a number of reasons: One of the table/column names that were set up in the configuration does not exist (because it was renamed or removed) or the credentials used by the BPS do not have sufficient privileges. Simply copy the SQL statement and execute it manually on the database with a regular database client that will show you the detailed error message returned by the DBMS. <syntaxhighlight> <biocase:diagnostic severity="WARNING"> The mandatory child element SourceInstitutionID could not be created [0 existing, but 1 required]. The element /DataSets/DataSet/Units/Unit was also dropped. </biocase:diagnostic> <biocase:diagnostic severity="WARNING"> The mandatory child element Units could not be created [0 existing, but 1 required]. The element /DataSets/DataSet was also dropped. </biocase:diagnostic> </syntaxhighlight> One problem caused the next one in this sequence of warnings: The mandatory ABCD element ''SourceInstitutionID '' was either not mapped or empty; therefore the respective record (''unit'') was dropped. Because this happened for all units in the dataset, all units were dropped, resulting in an empty ''Units'' sub tree. The second warning tells you what happened then: Because the ''Units'' element is mandatory for the ''Dataset'' element, the whole dataset was dropped. As a result, the response ABCD document was empty. == Debugging a BioCASe Installation == In case the debug output of the web service doesn’t help you to find the problem, or if the BPS behaves not as intended without displaying any error messages, you should have a look at the debug logs written by the BPS. To turn on the Debug logs, go to the System Administration page and set <small>Debug logs</small> to <small>True</small>. [[File:instConfigTool.png|500px]] Redo the step that didn’t work and open the ''log'' folder of your BioCASe installation. There are several global log files (pywrapper_request.log, pywrapper_error.log, webapp_debug.log, webapp_error.log) and a debug files per data source (e.g. debug_flora.log). Open these files and see if they can give you a clue to the problem. If you still can’t get it running, [[Special:Contact|contact the BioCASe team]]. 77cba1cb76e7f123dd3cc549134c732f033ddb01 0 24 369 332 2011-08-18T13:45:19Z JoergHoletschek 3 /* Creating a New Data Source */ wikitext text/x-wiki This tutorial explains in detail how to set up a datasource. == A BioCASe Data Source == Publishing a database with BioCASe means it will be made accessible through a BioCASe web service, which will be used later by the biodiversity network(s) you want to join to access to your data. For example, the GBIF Harvester will send a BioCASe Scan request to that web service to get a list of all scientific names kept in the database, and subsequently several BioCASe Search requests to load all records for these names. For each database you want to publish, you need to set up a BioCASe data source (do not mix that up with ODBC data sources on Windows machines). The resulting BioCASe web service is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source. So if you made your installation available at http://www.foobar.org/biocase during the installation process and set up a data source named Herbar, the URL of the BioCASe web service would be http://www.foobar.org/biocase/pywrapper.cgi?dsa=Herbar (data source names can be case sensitive, depending on your server’s operating system). == Creating a New Data Source == The start page of the BPS will always list the data sources currently configured, that means the web services offered by that BioCASe installation. On a fresh installation, it will be empty: [[File:instBiocaseStartpage.png|500px]] Creating a new data source can be done in the configuration tool. Choose <small>Config Tool</small> on the start page, then <small>System Administration</small>. Log in with the password you set during the installation process (if you didn’t change that yet, the default is ''ACDC''). In the configuration screen, the sections <small>Data Sources</small> and <small>Recycle Bin</small> can be used to create, remove und restore a data source: [[File:dsaCreateDatasourceEmpty.png|500px|border]] For now, we want to create a new, empty data source. Enter a name – without any spaces – in the text box and click <small>Create DSA</small> (remember that the datasource name is case sensitive on Linux machines). After a second, it will appear in the list of existing data sources with a clickable link. By doing this, you created the first BioCASe web service for your installation. Clicking <small>Remove</small> will uninstall this service again (putting it in the recycle bin, so you can restore it later). Tip: If you select <small>abcdmetadata</small> in the template drop-down list, your data source will be created with an ABCD mapping for all the metadata elements stored in the metadata table mentioned in the [[Preparation]] guide. So if you did use the templates in ''sampledb/abcdmetadata'' to create the metadata table in your database, you can use this template to spare the hassle of mapping this table and start right away with your occurrence data. == Setting up the database connection == Clicking on the data source you’ve just created will take you to the datasource configuration tool. Note that the <small>Schemas</small> section will be empty if you’ve created an empty data source or list two schemas (ABCD 1.2 and 2.06) if you created the datasource based on the <small>abcdmetadata</small> template: [[File:dsaOverview.png|450px]] The link in the <small>Database Connection</small> section will take you to the connection parameters dialog. Fill in the values for your database, press <small>Save</small> and the connection status should turn into a nice green <small>OK</small>. '''DBMS:''' Select the entry appropriate for your collection. The BPS can directly connect to database servers (MySQL, Postgres, DB2, SQL Server, and Oracle) and to Desktop-DBMS like Access and FoxPro through ODBC. ODBC is only available on Windows, so if you plan to use it, BioCASe must be installed on a Windows Server. '''Host:''' Enter the IP address of the database server. If you’re connecting through ODBC, you must leave this empty. Instead, use Windows’ ODBC Administrator to set up a System (!) datasource; the server or file name of the database will be stored in the ODBC configuration. '''Database:''' Enter the name of the ODBC datasource if you’re using ODBC, or the name of the database if you’re directly connecting to a server. '''User/Password:''' Database credentials to be used. As written in the [[Preparation]] guide, you should create a user with just read access for this. '''Encoding:''' The encoding of your database. If you’re not sure, leave the default, you can change it if you run into problems later. If you know your database is using a different encoding than latin-1 (which is typically the default for western languages), maybe UTF-8, select the appropriate entry in the drop-down list. [[File:dsaDBConnection.png|450px]] Once you’ve entered the correct values, press <small>Save</small>. Click <small>Overview</small> in the link list at the top of the page to return to the datasource configuration page. If the connection status doesn’t turn into <small>OK</small> (and doesn’t after you’ve returned to the overview page), please read the [[Debugging]] guide about how to trace the problem. == Setting up the Database Structure == Clicking on <small>Edit DB Structure</small> on the datasource overview page will show the DB structure setup. Make sure the connection status at the top of the page still is <small>OK</small>. In a first step, you will add all tables/views that hold information you want to publish with BioCASe. You can ignore all database tables storing only data you don’t plan to map to ABCD. Same holds true if you use cache tables (as explained in the [[Preparation]] guide): Only the cache tables need to be set up on the database structure page. In a second step, you will specify the foreign key references between these tables. === Adding Tables/Views with their Primary Keys === To add a table or view, just select it in the drop-down list. The name of the table will be filled in the <small>Alias</small> text box (you will use this alias later in the mapping process), feel free to change it into something that describes the contents of that tables (for example ''specimen'' for the root specimen table). Press <small>Add</small> to insert the table into the list. Before continuing to the next table, select the primary key in the list of attributes under <small>Primary Key Attribute(s)</small>. In case the data type is a string and not an integer, select <small>text</small> from the drop-down list. Press <small>Add</small> to add the primary key. If the primary key consists of several columns, you can add several primary key attributes. Redo these steps (adding the tables first, then the primary key/keys) for all tables you want to publish data from. Always remember to set the correct primary key column(s), even if they’re not defined as primary keys in the database.<br> '''<u>Important Note:</u>''' Tables without a primary key cannot be used by the Provider Software and will be removed upon saving!''' === Specifying Foreign Key References === Once you’ve added all tables, you can define the relations between these tables by defining foreign keys. In order to do this, go to the <small>Foreign Key(s)</small> column of the table that holds a foreign key reference to another table, select the alias of the referenced table and press <small>Add</small>. Select the attribute that stores the foreign key and change the data type if it is not integer, then press <small>Add</small>. You must specify as many attributes for the foreign keys as the primary key of the referenced table has, otherwise the Provider Software won’t be able to create a valid join between the tables. Also, they must be in the same order and have identical data types. Redo this for all tables until you’ve specified all foreign key references. If you’ve installed the Graphviz package, the Provider Software will create a graph visualizing the current database structure. Each table will be visible as a node with the primary key(s) listed. References between tables will be represented by arrows, the foreign key(s) listed in the referring table: [[File:dsaDatabaseStructure_AlgenEngels.png|800px]] === Saving, Editing an Existing Setup === You can change aliases of tables you’ve already added to the list. To do so, go to the <small>Alias</small> text box of that table and type in a different name. Upon exiting the text box, all references to this alias will be updated in the table setup; also the graph will reflect the changes. But caution: Schema mappings use these aliases to bind schema elements to tables and columns, if you change an alias you’ve already used in a schema mapping, you will break this binding! So it is advisable to decide for meaningful alias names and keep them stable once you’ve started the mapping process. Once you’ve added a table/view to the table list, you can’t change the underlying table, since this would invalidate all the primary and foreign key attributes. If you’ve added the wrong table, remove it by using the trash can symbol next to <small>Delete Alias</small> and add the correct table/view. If you’ve finished the table setup and the graph displays the model of your database correctly, save the configuration by pressing <small>Save</small> at the bottom of the page. This is important; navigating away from the page without saving will cause your changes to be lost (even though you will be asked for confirmation)! It is advisable to save the table setup every now and then, for example once you’ve added all tables and primary keys. If you edit an existing table setup and happen to realize you’ve messed it up, you can revert to the state when you’ve pressed <small>Save</small> for the last time by using the <small>Revert</small> button at the bottom. But be careful: All unsaved changes will be lost. '''<u>Important Note:</u>''' The retrieval of tables/view/column names is currently only supported for the following DBMS: Microsoft SQL Server, MySQL, Access, Postgres and FoxPro. If you use a different DBMS, you will see a simple text box instead of a drop-down-list, where you’ll have to type in the name of the table or column you want to add. a8c8503c89716991e32c703e9ecffb53098a80de 0 6 370 343 2011-08-18T15:26:31Z JoergHoletschek 3 /* Installing Additional Python Packages */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can a pain; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a Postgres and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] b0134e79c2a2e9446545eab65c8cc61da2914398 0 23 371 347 2011-08-23T09:26:12Z JoergHoletschek 3 wikitext text/x-wiki == Current Version 2.6 == ===Version 2.6.1 [upcoming]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler: Unary operator (isNull, isNotNull) are processed correctly now. ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.5.5 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. See example [http://ww3.bgbm.org/bps2/ScreenShots?action=AttachFile&do=get&target=config_tool4.gif here] * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 8a98877cbcad1913fa3a9535da8213512e6f9494 372 371 2011-08-23T09:27:25Z JoergHoletschek 3 /* Version 2.6.1 [upcoming] */ wikitext text/x-wiki == Current Version 2.6 == ===Version 2.6.1 [upcoming]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.5.5 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. See example [http://ww3.bgbm.org/bps2/ScreenShots?action=AttachFile&do=get&target=config_tool4.gif here] * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool bf9ca7afd8f4d361bf4bc06022d1bc3e471be4d1 373 372 2011-08-23T09:49:00Z JoergHoletschek 3 wikitext text/x-wiki == Current Version 2.6 == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.5.5 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. See example [http://ww3.bgbm.org/bps2/ScreenShots?action=AttachFile&do=get&target=config_tool4.gif here] * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 8bab862194a4313ffc6490ec9f2493c1c86b8276 374 373 2011-08-24T14:44:15Z JoergHoletschek 3 /* Version 2.5.4 [2010-09-14] */ wikitext text/x-wiki == Current Version 2.6 == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. See example [http://ww3.bgbm.org/bps2/ScreenShots?action=AttachFile&do=get&target=config_tool4.gif here] * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool f4058c821e538912596243792031a4f363369537 388 374 2011-11-01T13:13:59Z JoergHoletschek 3 wikitext text/x-wiki == Current Version 3 == ===Version 3.0RC === * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) ** Check for uniqueness of IDs in the list returned by first query only done when this is not done by a DISTINCT clause in query == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. See example [http://ww3.bgbm.org/bps2/ScreenShots?action=AttachFile&do=get&target=config_tool4.gif here] * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 456404fc2f7b7c2ff8c77ba48383a81c3bfe4598 389 388 2011-11-01T16:06:54Z JoergHoletschek 3 /* Version 3.0RC */ wikitext text/x-wiki == Current Version 3 == ===Version 3.0RC [2011-11-01]=== * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) ** Check for uniqueness of IDs in the list returned by first query only done when this is not done by a DISTINCT clause in query == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. See example [http://ww3.bgbm.org/bps2/ScreenShots?action=AttachFile&do=get&target=config_tool4.gif here] * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool eab087ccf72874aeed733a9204a2e748b9232ef8 390 389 2011-11-01T16:07:30Z JoergHoletschek 3 /* Version 3.0RC [2011-11-01] */ wikitext text/x-wiki == Current Version 3 == ===Version 3.0RC [2011-11-01]=== * New Archiving feature * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) ** Check for uniqueness of IDs in the list returned by first query only done when this is not done by a DISTINCT clause in query == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. See example [http://ww3.bgbm.org/bps2/ScreenShots?action=AttachFile&do=get&target=config_tool4.gif here] * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 611fb0c2202fd086f7c4d33476ff496b3cced56c 391 390 2011-11-01T16:08:33Z JoergHoletschek 3 /* Version 3.0RC [2011-11-01] */ wikitext text/x-wiki == Current Version 3 == ===Version 3.0RC [2011-11-01]=== * New Archiving feature * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. See example [http://ww3.bgbm.org/bps2/ScreenShots?action=AttachFile&do=get&target=config_tool4.gif here] * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 68342dc6021740eb3bdebde466ffbc112ab32934 392 391 2011-11-01T16:16:34Z JoergHoletschek 3 /* Version 2.3.0 [2005-11-03] */ wikitext text/x-wiki == Current Version 3 == ===Version 3.0RC [2011-11-01]=== * New Archiving feature * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 1fdf4bccb0dca76b7fd8d5bc523f39458cefdfc9 0 52 377 340 2011-09-06T16:18:06Z JoergHoletschek 3 /* In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I cannot find the installation package of the database connectivity module for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later of psycopg2. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation to the ''config'' folder of your new installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other only by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. == Trouble Shooting == ===I’ve forgot the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. 07337b49ffbede42c61ffd055ae5e4794fdfba5c 378 377 2011-09-13T09:28:25Z JoergHoletschek 3 /* I want to update my BioCASe installation. How do I do this? */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I cannot find the installation package of the database connectivity module for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later of psycopg2. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation to the ''config'' folder of your new installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. You can then edit new installation and test the web services. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other only by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. == Trouble Shooting == ===I’ve forgot the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. 6d9f9958a639fd4c577c9efc88cbbae77e4326e5 379 378 2011-09-13T09:31:29Z JoergHoletschek 3 /* I’ve forgot the password of my BioCASe installation. What should I do? */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I cannot find the installation package of the database connectivity module for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later of psycopg2. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation to the ''config'' folder of your new installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. You can then edit new installation and test the web services. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other only by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. == Trouble Shooting == ===I’ve forgotten the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. a02d13a5353cd6494dbdca04dd5fd88105219166 393 379 2011-11-07T12:47:58Z JoergHoletschek 3 /* Installation */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I cannot find the installation package of the database connectivity module for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later of psycopg2. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation to the ''config'' folder of your new installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. You can then edit new installation and test the web services. ===When I try to create an archive, I get a "Permission denied" error. Why is that?=== For creating archives, you need to grant write permissions for the folders ''archive'' and ''www/downloads'' of your BioCASe installation to Python. You can check if permissions are set correctly in the test libs page (Start --> Utilities --> Test Libs). == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other only by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. == Trouble Shooting == ===I’ve forgotten the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. 8e790ebd1bec132f128e6613ba34c2354b7b8213 0 22 383 346 2011-09-21T12:07:02Z JoergHoletschek 3 wikitext text/x-wiki This page lists the most commonly used ABCD 2.06 concepts. The ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a full list of ABCD concepts [http://wiki.tdwg.org/twiki/bin/view/ABCD/AbcdConcepts here]. == Mandatory == List of mandatory concepts. Without them being mapped the Provider Software will not be able to create valid ABCD document: <pre> DataSets └─ DataSet ├─ ContentContacts │ └─ ContentContact │ └─ Name ├─ TechnicalContacts │ └─ TechnicalContact │ └─ Name ├─ Metadata │ ├─ Description │ │ └─ Representation │ │ ├─ @language │ │ └─ Title │ └─ RevisionData │ └─ DateModified └─ Units └─ Unit ├─ SourceInstitutionID ├─ SourceID └─ UnitID </pre> == Highly Recommended == List of recommended ABCD concepts. If you can provide this data, please do so, they're commonly used entry points to your data. <pre> DataSets └─ DataSet ├─ ContentContacts │ └─ ContentContact │ ├─ Email │ └─ Address ├─ TechnicalContacts │ └─ TechnicalContact │ ├─ Email │ └─ Address ├─ Metadata │ ├─ Description │ │ └─ Representation │ │ └─ Details │ └─ Owners │ └─ Owner │ ├─ LogoURI │ ├─ URIs │ │ └─ URL │ ├─ Person │ │ └─ FullName │ └─ Organisation │ └─ Name │ └─ Representation │ └─ Text └─ Units └─ Unit ├─ SourceInstitutionID ├─ SourceID └─ UnitID </pre> Unitdata * /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString * /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/HigherTaxa/HigherTaxon/HigherTaxonName * /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/HigherTaxa/HigherTaxon/HigherTaxonRank * /DataSets/DataSet/Units/Unit/Identifications/Identification/PreferredFlag * /DataSets/DataSet/Units/Unit/RecordBasis * /DataSets/DataSet/Units/Unit/Gathering/DateTime/ISODateTimeBegin * /DataSets/DataSet/Units/Unit/Gathering/Agents/GatheringAgent/Person/FullName * /DataSets/DataSet/Units/Unit/Gathering/LocalityText * /DataSets/DataSet/Units/Unit/Gathering/Country/ISO3166Code * /DataSets/DataSet/Units/Unit/Gathering/Country/Name * /DataSets/DataSet/Units/Unit/Gathering/SiteCoordinateSets/SiteCoordinates/CoordinatesLatLong/LongitudeDecimal */DataSets/DataSet/Units/Unit/Gathering/SiteCoordinateSets/SiteCoordinates/CoordinatesLatLong/LatitudeDecimal == Commonly Used == List of further commonly mapped ABCD concepts. Metadata * /DataSets/DataSet/IPRStatements/TermsOfUseStatements/TermsOfUse/Text * /DataSets/DataSet/IPRStatements/Copyrights/Copyright/Text * /DataSets/DataSet/IPRStatements/IPRDeclarations/IPRDeclaration/Text * /DataSets/DataSet/IPRStatements/Licenses/License/Text * /DataSets/DataSet/IPRStatements/Disclaimers/Disclaimer/Text * /DataSets/DataSet/IPRStatements/Citations/Citation/Text * /DataSets/DataSet/IPRStatements/Acknowledgements/Acknowledgement/Text Unitdata * /DataSets/DataSet/Units/Unit/DateLastEdited * /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/InformalNameString * /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/InformalNameString[@language] * /DataSets/DataSet/Units/Unit/Identifications/Identification/Identifiers/Identifier/IdentifierPersonName/FullName * /DataSets/DataSet/Units/Unit/Identifications/Identification/Date/ISODateTimeBegin * /DataSets/DataSet/Units/Unit/KindOfUnit * /DataSets/DataSet/Units/Unit/SpecimenUnit/NomenclaturalTypeDesignations/NomenclaturalTypeDesignation/TypifiedName/FullScientificNameString * /DataSets/DataSet/Units/Unit/SpecimenUnit/NomenclaturalTypeDesignations/NomenclaturalTypeDesignation/TypeStatus */DataSets/DataSet/Units/Unit/SpecimenUnit/NomenclaturalTypeDesignations/NomenclaturalTypeDesignation/NomenclaturalReference/TitleCitation * /DataSets/DataSet/Units/Unit/SpecimenUnit/NomenclaturalTypeDesignations/NomenclaturalTypeDesignation/NomenclaturalReference/CitationDetail * /DataSets/DataSet/Units/Unit/MultiMediaObjects/MultiMediaObject/FileURI * /DataSets/DataSet/Units/Unit/MultiMediaObjects/MultiMediaObject/Comment * /DataSets/DataSet/Units/Unit/MultiMediaObjects/MultiMediaObject/Context * /DataSets/DataSet/Units/Unit/Gathering/Project/ProjectTitle * /DataSets/DataSet/Units/Unit/Gathering/NamedAreas/NamedArea/AreaName * /DataSets/DataSet/Units/Unit/Gathering/NamedAreas/NamedArea/AreaClass * /DataSets/DataSet/Units/Unit/Gathering/Altitude/MeasurementOrFactAtomised/LowerValue * /DataSets/DataSet/Units/Unit/Gathering/Altitude/MeasurementOrFactAtomised/UnitOfMeasurement * /DataSets/DataSet/Units/Unit/Gathering/Depth/MeasurementOrFactAtomised/LowerValue * /DataSets/DataSet/Units/Unit/Gathering/Depth/MeasurementOrFactAtomised/UnitOfMeasurement * /DataSets/DataSet/Units/Unit/Gathering/Biotope/Name * /DataSets/DataSet/Units/Unit/CollectorsFieldNumber * /DataSets/DataSet/Units/Unit/Sex * /DataSets/DataSet/Units/Unit/Age * /DataSets/DataSet/Units/Unit/Notes * /DataSets/DataSet/Units/Unit/RecordURI d1a51a7c1726a1b26c6ac52970796fa7361528ff 384 383 2011-09-21T12:21:20Z JoergHoletschek 3 /* Highly Recommended */ wikitext text/x-wiki This page lists the most commonly used ABCD 2.06 concepts. The ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a full list of ABCD concepts [http://wiki.tdwg.org/twiki/bin/view/ABCD/AbcdConcepts here]. == Mandatory == List of mandatory concepts. Without them being mapped the Provider Software will not be able to create valid ABCD document: <pre> DataSets └─ DataSet ├─ ContentContacts │ └─ ContentContact │ └─ Name ├─ TechnicalContacts │ └─ TechnicalContact │ └─ Name ├─ Metadata │ ├─ Description │ │ └─ Representation │ │ ├─ @language │ │ └─ Title │ └─ RevisionData │ └─ DateModified └─ Units └─ Unit ├─ SourceInstitutionID ├─ SourceID └─ UnitID </pre> == Highly Recommended == List of recommended ABCD concepts. If you can provide this data, please do so, they're commonly used entry points to your data. <pre> DataSets └─ DataSet ├─ ContentContacts │ └─ ContentContact │ ├─ Email │ └─ Address ├─ TechnicalContacts │ └─ TechnicalContact │ ├─ Email │ └─ Address ├─ Metadata │ ├─ Description │ │ └─ Representation │ │ └─ Details │ └─ Owners │ └─ Owner │ ├─ LogoURI │ ├─ URIs │ │ └─ URL │ ├─ Person │ │ └─ FullName │ └─ Organisation │ └─ Name │ └─ Representation │ └─ Text └─ Units └─ Unit ├─ Identifications │ └─ Identification │ └─ Result │ ├─ @PreferredFlag │ ├─ TaxonIdentified │ │ └─ ScientificName │ │ └─ FullScientificNameString │ └─ HigherTaxa │ └─ HigherTaxon │ ├─ HigherTaxonName │ └─ HigherTaxonRank ├─ RecordBasis └─ Gathering ├─ DateTime │ └─ ISODateTimeBegin ├─ Agents │ └─ GatheringAgent │ └─ Person │ └─ FullName ├─ LocalityText ├─ Country │ ├─ ISO3166Code │ └─ Name └─ SiteCoordinateSets └─ SiteCoordinates └─ CoordinatesLatLong ├─ LongitudeDecimal └─ LatitudeDecimal </pre> == Commonly Used == List of further commonly mapped ABCD concepts. Metadata * /DataSets/DataSet/IPRStatements/TermsOfUseStatements/TermsOfUse/Text * /DataSets/DataSet/IPRStatements/Copyrights/Copyright/Text * /DataSets/DataSet/IPRStatements/IPRDeclarations/IPRDeclaration/Text * /DataSets/DataSet/IPRStatements/Licenses/License/Text * /DataSets/DataSet/IPRStatements/Disclaimers/Disclaimer/Text * /DataSets/DataSet/IPRStatements/Citations/Citation/Text * /DataSets/DataSet/IPRStatements/Acknowledgements/Acknowledgement/Text Unitdata * /DataSets/DataSet/Units/Unit/DateLastEdited * /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/InformalNameString * /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/InformalNameString[@language] * /DataSets/DataSet/Units/Unit/Identifications/Identification/Identifiers/Identifier/IdentifierPersonName/FullName * /DataSets/DataSet/Units/Unit/Identifications/Identification/Date/ISODateTimeBegin * /DataSets/DataSet/Units/Unit/KindOfUnit * /DataSets/DataSet/Units/Unit/SpecimenUnit/NomenclaturalTypeDesignations/NomenclaturalTypeDesignation/TypifiedName/FullScientificNameString * /DataSets/DataSet/Units/Unit/SpecimenUnit/NomenclaturalTypeDesignations/NomenclaturalTypeDesignation/TypeStatus */DataSets/DataSet/Units/Unit/SpecimenUnit/NomenclaturalTypeDesignations/NomenclaturalTypeDesignation/NomenclaturalReference/TitleCitation * /DataSets/DataSet/Units/Unit/SpecimenUnit/NomenclaturalTypeDesignations/NomenclaturalTypeDesignation/NomenclaturalReference/CitationDetail * /DataSets/DataSet/Units/Unit/MultiMediaObjects/MultiMediaObject/FileURI * /DataSets/DataSet/Units/Unit/MultiMediaObjects/MultiMediaObject/Comment * /DataSets/DataSet/Units/Unit/MultiMediaObjects/MultiMediaObject/Context * /DataSets/DataSet/Units/Unit/Gathering/Project/ProjectTitle * /DataSets/DataSet/Units/Unit/Gathering/NamedAreas/NamedArea/AreaName * /DataSets/DataSet/Units/Unit/Gathering/NamedAreas/NamedArea/AreaClass * /DataSets/DataSet/Units/Unit/Gathering/Altitude/MeasurementOrFactAtomised/LowerValue * /DataSets/DataSet/Units/Unit/Gathering/Altitude/MeasurementOrFactAtomised/UnitOfMeasurement * /DataSets/DataSet/Units/Unit/Gathering/Depth/MeasurementOrFactAtomised/LowerValue * /DataSets/DataSet/Units/Unit/Gathering/Depth/MeasurementOrFactAtomised/UnitOfMeasurement * /DataSets/DataSet/Units/Unit/Gathering/Biotope/Name * /DataSets/DataSet/Units/Unit/CollectorsFieldNumber * /DataSets/DataSet/Units/Unit/Sex * /DataSets/DataSet/Units/Unit/Age * /DataSets/DataSet/Units/Unit/Notes * /DataSets/DataSet/Units/Unit/RecordURI 9c6b94e4f907cb3c9bc52af27bb2ad220e37e6dd 385 384 2011-09-21T14:22:19Z JoergHoletschek 3 wikitext text/x-wiki This page lists the most commonly used ABCD 2.06 concepts. The ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a full list of ABCD concepts [http://wiki.tdwg.org/twiki/bin/view/ABCD/AbcdConcepts here]. == Mandatory == List of mandatory concepts. Without them being mapped the Provider Software will not be able to create valid ABCD document: <pre> DataSets └─ DataSet ├─ ContentContacts │ └─ ContentContact │ └─ Name ├─ TechnicalContacts │ └─ TechnicalContact │ └─ Name ├─ Metadata │ ├─ Description │ │ └─ Representation │ │ ├─ @language │ │ └─ Title │ └─ RevisionData │ └─ DateModified └─ Units └─ Unit ├─ SourceInstitutionID ├─ SourceID └─ UnitID </pre> == Highly Recommended == List of recommended ABCD concepts. If you can provide this data, please do so, they're commonly used entry points to your data. <pre> DataSets └─ DataSet ├─ ContentContacts │ └─ ContentContact │ ├─ Email │ └─ Address ├─ TechnicalContacts │ └─ TechnicalContact │ ├─ Email │ └─ Address ├─ Metadata │ ├─ Description │ │ └─ Representation │ │ └─ Details │ └─ Owners │ └─ Owner │ ├─ LogoURI │ ├─ URIs │ │ └─ URL │ ├─ Person │ │ └─ FullName │ └─ Organisation │ └─ Name │ └─ Representation │ └─ Text └─ Units └─ Unit ├─ Identifications │ └─ Identification │ └─ Result │ ├─ @PreferredFlag │ ├─ TaxonIdentified │ │ └─ ScientificName │ │ └─ FullScientificNameString │ └─ HigherTaxa │ └─ HigherTaxon │ ├─ HigherTaxonName │ └─ HigherTaxonRank ├─ RecordBasis └─ Gathering ├─ DateTime │ └─ ISODateTimeBegin ├─ Agents │ └─ GatheringAgent │ └─ Person │ └─ FullName ├─ LocalityText ├─ Country │ ├─ ISO3166Code │ └─ Name └─ SiteCoordinateSets └─ SiteCoordinates └─ CoordinatesLatLong ├─ LongitudeDecimal └─ LatitudeDecimal </pre> == Commonly Used == List of further commonly mapped ABCD concepts. <pre> DataSets └─ DataSet ├─ IPRStatements │ ├─ TermsOfUseStatements │ │ └─ TermsOfUse │ │ └─ Text │ ├─ Copyrights │ │ └─ Copyright │ │ └─ Text │ ├─ IPRDeclarations │ │ └─ IPRDeclaration │ │ └─ Text │ ├─ Licenses │ │ └─ License │ │ └─ Text │ ├─ Disclaimers │ │ └─ Disclaimer │ │ └─ Text │ ├─ Citations │ │ └─ Citation │ │ └─ Text │ └─ Acknowledgements │ └─ Acknowledgement │ └─ Text └─ Units └─ Unit ├─ Identifications │ └─ Identification │ └─ Result │ ├─ @PreferredFlag │ ├─ TaxonIdentified │ │ └─ ScientificName │ │ └─ FullScientificNameString │ └─ HigherTaxa │ └─ HigherTaxon │ ├─ HigherTaxonName │ └─ HigherTaxonRank ├─ RecordBasis └─ Gathering ├─ DateTime │ └─ ISODateTimeBegin ├─ Agents │ └─ GatheringAgent │ └─ Person │ └─ FullName ├─ LocalityText ├─ Country │ ├─ ISO3166Code │ └─ Name └─ SiteCoordinateSets └─ SiteCoordinates └─ CoordinatesLatLong ├─ LongitudeDecimal └─ LatitudeDecimal </pre> Unitdata * /DataSets/DataSet/Units/Unit/DateLastEdited * /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/InformalNameString * /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/InformalNameString[@language] * /DataSets/DataSet/Units/Unit/Identifications/Identification/Identifiers/Identifier/IdentifierPersonName/FullName * /DataSets/DataSet/Units/Unit/Identifications/Identification/Date/ISODateTimeBegin * /DataSets/DataSet/Units/Unit/KindOfUnit * /DataSets/DataSet/Units/Unit/SpecimenUnit/NomenclaturalTypeDesignations/NomenclaturalTypeDesignation/TypifiedName/FullScientificNameString * /DataSets/DataSet/Units/Unit/SpecimenUnit/NomenclaturalTypeDesignations/NomenclaturalTypeDesignation/TypeStatus */DataSets/DataSet/Units/Unit/SpecimenUnit/NomenclaturalTypeDesignations/NomenclaturalTypeDesignation/NomenclaturalReference/TitleCitation * /DataSets/DataSet/Units/Unit/SpecimenUnit/NomenclaturalTypeDesignations/NomenclaturalTypeDesignation/NomenclaturalReference/CitationDetail * /DataSets/DataSet/Units/Unit/MultiMediaObjects/MultiMediaObject/FileURI * /DataSets/DataSet/Units/Unit/MultiMediaObjects/MultiMediaObject/Comment * /DataSets/DataSet/Units/Unit/MultiMediaObjects/MultiMediaObject/Context * /DataSets/DataSet/Units/Unit/Gathering/Project/ProjectTitle * /DataSets/DataSet/Units/Unit/Gathering/NamedAreas/NamedArea/AreaName * /DataSets/DataSet/Units/Unit/Gathering/NamedAreas/NamedArea/AreaClass * /DataSets/DataSet/Units/Unit/Gathering/Altitude/MeasurementOrFactAtomised/LowerValue * /DataSets/DataSet/Units/Unit/Gathering/Altitude/MeasurementOrFactAtomised/UnitOfMeasurement * /DataSets/DataSet/Units/Unit/Gathering/Depth/MeasurementOrFactAtomised/LowerValue * /DataSets/DataSet/Units/Unit/Gathering/Depth/MeasurementOrFactAtomised/UnitOfMeasurement * /DataSets/DataSet/Units/Unit/Gathering/Biotope/Name * /DataSets/DataSet/Units/Unit/CollectorsFieldNumber * /DataSets/DataSet/Units/Unit/Sex * /DataSets/DataSet/Units/Unit/Age * /DataSets/DataSet/Units/Unit/Notes * /DataSets/DataSet/Units/Unit/RecordURI bc458f1c516238bfa3edcf145ab0f89fde2411d0 386 385 2011-09-21T14:25:01Z JoergHoletschek 3 /* Commonly Used */ wikitext text/x-wiki This page lists the most commonly used ABCD 2.06 concepts. The ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a full list of ABCD concepts [http://wiki.tdwg.org/twiki/bin/view/ABCD/AbcdConcepts here]. == Mandatory == List of mandatory concepts. Without them being mapped the Provider Software will not be able to create valid ABCD document: <pre> DataSets └─ DataSet ├─ ContentContacts │ └─ ContentContact │ └─ Name ├─ TechnicalContacts │ └─ TechnicalContact │ └─ Name ├─ Metadata │ ├─ Description │ │ └─ Representation │ │ ├─ @language │ │ └─ Title │ └─ RevisionData │ └─ DateModified └─ Units └─ Unit ├─ SourceInstitutionID ├─ SourceID └─ UnitID </pre> == Highly Recommended == List of recommended ABCD concepts. If you can provide this data, please do so, they're commonly used entry points to your data. <pre> DataSets └─ DataSet ├─ ContentContacts │ └─ ContentContact │ ├─ Email │ └─ Address ├─ TechnicalContacts │ └─ TechnicalContact │ ├─ Email │ └─ Address ├─ Metadata │ ├─ Description │ │ └─ Representation │ │ └─ Details │ └─ Owners │ └─ Owner │ ├─ LogoURI │ ├─ URIs │ │ └─ URL │ ├─ Person │ │ └─ FullName │ └─ Organisation │ └─ Name │ └─ Representation │ └─ Text └─ Units └─ Unit ├─ Identifications │ └─ Identification │ └─ Result │ ├─ @PreferredFlag │ ├─ TaxonIdentified │ │ └─ ScientificName │ │ └─ FullScientificNameString │ └─ HigherTaxa │ └─ HigherTaxon │ ├─ HigherTaxonName │ └─ HigherTaxonRank ├─ RecordBasis └─ Gathering ├─ DateTime │ └─ ISODateTimeBegin ├─ Agents │ └─ GatheringAgent │ └─ Person │ └─ FullName ├─ LocalityText ├─ Country │ ├─ ISO3166Code │ └─ Name └─ SiteCoordinateSets └─ SiteCoordinates └─ CoordinatesLatLong ├─ LongitudeDecimal └─ LatitudeDecimal </pre> == Commonly Used == List of further commonly mapped ABCD concepts. <pre> DataSets └─ DataSet ├─ Metadata │ └─ IPRStatements │ ├─ TermsOfUseStatements │ │ └─ TermsOfUse │ │ └─ Text │ ├─ Copyrights │ │ └─ Copyright │ │ └─ Text │ ├─ IPRDeclarations │ │ └─ IPRDeclaration │ │ └─ Text │ ├─ Licenses │ │ └─ License │ │ └─ Text │ ├─ Disclaimers │ │ └─ Disclaimer │ │ └─ Text │ ├─ Citations │ │ └─ Citation │ │ └─ Text │ └─ Acknowledgements │ └─ Acknowledgement │ └─ Text └─ Units └─ Unit ├─ Identifications │ └─ Identification │ └─ Result │ ├─ @PreferredFlag │ ├─ TaxonIdentified │ │ └─ ScientificName │ │ └─ FullScientificNameString │ └─ HigherTaxa │ └─ HigherTaxon │ ├─ HigherTaxonName │ └─ HigherTaxonRank ├─ RecordBasis └─ Gathering ├─ DateTime │ └─ ISODateTimeBegin ├─ Agents │ └─ GatheringAgent │ └─ Person │ └─ FullName ├─ LocalityText ├─ Country │ ├─ ISO3166Code │ └─ Name └─ SiteCoordinateSets └─ SiteCoordinates └─ CoordinatesLatLong ├─ LongitudeDecimal └─ LatitudeDecimal </pre> Unitdata * /DataSets/DataSet/Units/Unit/DateLastEdited * /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/InformalNameString * /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/InformalNameString[@language] * /DataSets/DataSet/Units/Unit/Identifications/Identification/Identifiers/Identifier/IdentifierPersonName/FullName * /DataSets/DataSet/Units/Unit/Identifications/Identification/Date/ISODateTimeBegin * /DataSets/DataSet/Units/Unit/KindOfUnit * /DataSets/DataSet/Units/Unit/SpecimenUnit/NomenclaturalTypeDesignations/NomenclaturalTypeDesignation/TypifiedName/FullScientificNameString * /DataSets/DataSet/Units/Unit/SpecimenUnit/NomenclaturalTypeDesignations/NomenclaturalTypeDesignation/TypeStatus */DataSets/DataSet/Units/Unit/SpecimenUnit/NomenclaturalTypeDesignations/NomenclaturalTypeDesignation/NomenclaturalReference/TitleCitation * /DataSets/DataSet/Units/Unit/SpecimenUnit/NomenclaturalTypeDesignations/NomenclaturalTypeDesignation/NomenclaturalReference/CitationDetail * /DataSets/DataSet/Units/Unit/MultiMediaObjects/MultiMediaObject/FileURI * /DataSets/DataSet/Units/Unit/MultiMediaObjects/MultiMediaObject/Comment * /DataSets/DataSet/Units/Unit/MultiMediaObjects/MultiMediaObject/Context * /DataSets/DataSet/Units/Unit/Gathering/Project/ProjectTitle * /DataSets/DataSet/Units/Unit/Gathering/NamedAreas/NamedArea/AreaName * /DataSets/DataSet/Units/Unit/Gathering/NamedAreas/NamedArea/AreaClass * /DataSets/DataSet/Units/Unit/Gathering/Altitude/MeasurementOrFactAtomised/LowerValue * /DataSets/DataSet/Units/Unit/Gathering/Altitude/MeasurementOrFactAtomised/UnitOfMeasurement * /DataSets/DataSet/Units/Unit/Gathering/Depth/MeasurementOrFactAtomised/LowerValue * /DataSets/DataSet/Units/Unit/Gathering/Depth/MeasurementOrFactAtomised/UnitOfMeasurement * /DataSets/DataSet/Units/Unit/Gathering/Biotope/Name * /DataSets/DataSet/Units/Unit/CollectorsFieldNumber * /DataSets/DataSet/Units/Unit/Sex * /DataSets/DataSet/Units/Unit/Age * /DataSets/DataSet/Units/Unit/Notes * /DataSets/DataSet/Units/Unit/RecordURI 2c192a8ea0e9207b2c4cd2da2d8b0e96f2b6873a 387 386 2011-09-21T14:39:59Z JoergHoletschek 3 /* Commonly Used */ wikitext text/x-wiki This page lists the most commonly used ABCD 2.06 concepts. The ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a full list of ABCD concepts [http://wiki.tdwg.org/twiki/bin/view/ABCD/AbcdConcepts here]. == Mandatory == List of mandatory concepts. Without them being mapped the Provider Software will not be able to create valid ABCD document: <pre> DataSets └─ DataSet ├─ ContentContacts │ └─ ContentContact │ └─ Name ├─ TechnicalContacts │ └─ TechnicalContact │ └─ Name ├─ Metadata │ ├─ Description │ │ └─ Representation │ │ ├─ @language │ │ └─ Title │ └─ RevisionData │ └─ DateModified └─ Units └─ Unit ├─ SourceInstitutionID ├─ SourceID └─ UnitID </pre> == Highly Recommended == List of recommended ABCD concepts. If you can provide this data, please do so, they're commonly used entry points to your data. <pre> DataSets └─ DataSet ├─ ContentContacts │ └─ ContentContact │ ├─ Email │ └─ Address ├─ TechnicalContacts │ └─ TechnicalContact │ ├─ Email │ └─ Address ├─ Metadata │ ├─ Description │ │ └─ Representation │ │ └─ Details │ └─ Owners │ └─ Owner │ ├─ LogoURI │ ├─ URIs │ │ └─ URL │ ├─ Person │ │ └─ FullName │ └─ Organisation │ └─ Name │ └─ Representation │ └─ Text └─ Units └─ Unit ├─ Identifications │ └─ Identification │ └─ Result │ ├─ @PreferredFlag │ ├─ TaxonIdentified │ │ └─ ScientificName │ │ └─ FullScientificNameString │ └─ HigherTaxa │ └─ HigherTaxon │ ├─ HigherTaxonName │ └─ HigherTaxonRank ├─ RecordBasis └─ Gathering ├─ DateTime │ └─ ISODateTimeBegin ├─ Agents │ └─ GatheringAgent │ └─ Person │ └─ FullName ├─ LocalityText ├─ Country │ ├─ ISO3166Code │ └─ Name └─ SiteCoordinateSets └─ SiteCoordinates └─ CoordinatesLatLong ├─ LongitudeDecimal └─ LatitudeDecimal </pre> == Commonly Used == List of further commonly mapped ABCD concepts. <pre> DataSets └─ DataSet ├─ Metadata │ └─ IPRStatements │ ├─ TermsOfUseStatements │ │ └─ TermsOfUse │ │ └─ Text │ ├─ Copyrights │ │ └─ Copyright │ │ └─ Text │ ├─ IPRDeclarations │ │ └─ IPRDeclaration │ │ └─ Text │ ├─ Licenses │ │ └─ License │ │ └─ Text │ ├─ Disclaimers │ │ └─ Disclaimer │ │ └─ Text │ ├─ Citations │ │ └─ Citation │ │ └─ Text │ └─ Acknowledgements │ └─ Acknowledgement │ └─ Text └─ Units └─ Unit ├─ DateLastEdited ├─ Identifications │ └─ Identification │ ├─ Date │ │ └─ ISODateTimeBegin │ └─ Identifiers │ └─ Identifier │ └─ IdentifierPersonName │ └─ FullName ├─ KindOfUnit ├─ CollectorsFieldNumber ├─ Sex ├─ Age ├─ Notes ├─ RecordURI ├─ MultiMediaObjects │ └─ MultiMediaObject │ ├─ FileURI │ ├─ Comment │ └─ Context ├─ SpecimenUnit │ └─ NomenclaturalTypeDesignations │ └─ NomenclaturalTypeDesignation │ ├─ TypeStatus │ ├─ TypifiedName │ │ └─ FullScientificNameString │ └─ NomenclaturalReference │ ├─ TitleCitation │ └─ CitationDetail └─ Gathering ├─ Project │ └─ ProjectTitle ├─ Biotope │ └─ Name ├─ Altitude │ └─ MeasurementOrFactAtomised │ ├─ LowerValue │ └─ UnitOfMeasurement ├─ Depth │ └─ MeasurementOrFactAtomised │ ├─ LowerValue │ └─ UnitOfMeasurement └─ NamedAreas └─ NamedArea ├─ AreaName └─ AreaClass </pre> 713c6ee80d5398f062518fcdd902437bf9940738 0 80 394 2011-11-15T10:13:09Z JoergHoletschek 3 Created page with "blubb" wikitext text/x-wiki blubb 71c4025dd9897b364f3ebbb42c484ff43d00791c 397 394 2011-11-15T14:46:06Z JoergHoletschek 3 wikitext text/x-wiki With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: XML Archives simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. DarwinCore Archives (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the TDWG site. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Archives using the ABCD 2.06 schema (dubbed as “ABCD dumps”) can be converted into one or several DarwinCore Archives in a subsequent step. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the DarwinCore data standard, which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than the ABCD dump it was created from. Creating XML Archives Once you’ve finished mapping a schema (see here how to do that) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the “Archive” link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. (bild) In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. “Create a new Archive” allows you to create a new archive for each schema mapped. Simply select the schema and press “Create Archive” (leave the defaults unless you have a good reason to change them). During the archiving process, the “Log” will display messages indicating the progress: (bild) You can always cancel the archiving process by pressing “Cancel” next to the respective entry listed under “in progress”. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing “Show full log” will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the “Finished” list, if it’s still being processed, under “In progress”. Pressing the “Info” link next to an entry will display the log again. “Download” and “Remove” will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. There are four parameters for customizing the archiving process: Destination schema: Data schema that will be used for storing the published records in the archive. Paging size: Number of records stored in each document. The default 1,000 should be suitable for most cases. Important note: Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. Max number of consecutive errors: The default for this is “1”. That means that if an error occurs during data retrieval, or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. Archive file name: Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. Creating a DarwinCore Archive 37f56ee8ee4add40b4e2571e0d7699891c4277ae 398 397 2011-11-15T14:50:49Z JoergHoletschek 3 wikitext text/x-wiki With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: XML Archives simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. DarwinCore Archives (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Archives using the ABCD 2.06 schema (dubbed as “ABCD dumps”) can be converted into one or several DarwinCore Archives in a subsequent step. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than the ABCD dump it was created from. ==XML Archives== Once you’ve finished mapping a schema (see here how to do that) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the “Archive” link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. (bild) In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. “Create a new Archive” allows you to create a new archive for each schema mapped. Simply select the schema and press “Create Archive” (leave the defaults unless you have a good reason to change them). During the archiving process, the “Log” will display messages indicating the progress: (bild) You can always cancel the archiving process by pressing “Cancel” next to the respective entry listed under “in progress”. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing “Show full log” will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the “Finished” list, if it’s still being processed, under “In progress”. Pressing the “Info” link next to an entry will display the log again. “Download” and “Remove” will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. There are four parameters for customizing the archiving process: Destination schema: Data schema that will be used for storing the published records in the archive. Paging size: Number of records stored in each document. The default 1,000 should be suitable for most cases. Important note: Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. Max number of consecutive errors: The default for this is “1”. That means that if an error occurs during data retrieval, or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. Archive file name: Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== 493afd0cbe1fe67b9130d40cbb04c05192c3453f 399 398 2011-11-15T14:56:30Z JoergHoletschek 3 /* XML Archives */ wikitext text/x-wiki With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: XML Archives simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. DarwinCore Archives (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Archives using the ABCD 2.06 schema (dubbed as “ABCD dumps”) can be converted into one or several DarwinCore Archives in a subsequent step. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than the ABCD dump it was created from. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. (bild) In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create a new Archive<small> allows you to create a new archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: (bild) You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. There are four parameters for customizing the archiving process: Destination schema: Data schema that will be used for storing the published records in the archive. Paging size: Number of records stored in each document. The default 1,000 should be suitable for most cases. Important note: Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. Max number of consecutive errors: The default for this is “1”. That means that if an error occurs during data retrieval, or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. Archive file name: Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== 8fc75d0d1359efd973b20cf9da2509bc3c96497a 400 399 2011-11-15T14:56:49Z JoergHoletschek 3 /* XML Archives */ wikitext text/x-wiki With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: XML Archives simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. DarwinCore Archives (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Archives using the ABCD 2.06 schema (dubbed as “ABCD dumps”) can be converted into one or several DarwinCore Archives in a subsequent step. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than the ABCD dump it was created from. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. (bild) In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create a new Archive</small> allows you to create a new archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: (bild) You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. There are four parameters for customizing the archiving process: Destination schema: Data schema that will be used for storing the published records in the archive. Paging size: Number of records stored in each document. The default 1,000 should be suitable for most cases. Important note: Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. Max number of consecutive errors: The default for this is “1”. That means that if an error occurs during data retrieval, or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. Archive file name: Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== 72b6c9f9c818df0fe2759be0d9cc0e62a6107075 401 400 2011-11-15T15:01:14Z JoergHoletschek 3 /* XML Archives */ wikitext text/x-wiki With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: XML Archives simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. DarwinCore Archives (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Archives using the ABCD 2.06 schema (dubbed as “ABCD dumps”) can be converted into one or several DarwinCore Archives in a subsequent step. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than the ABCD dump it was created from. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. (bild) In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create a new Archive</small> allows you to create a new archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: (bild) You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval, or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== 16a7bacba6a9d3dfa7e720d450df2468b47c3fdd 402 401 2011-11-15T15:02:19Z JoergHoletschek 3 /* XML Archives */ wikitext text/x-wiki With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: XML Archives simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. DarwinCore Archives (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Archives using the ABCD 2.06 schema (dubbed as “ABCD dumps”) can be converted into one or several DarwinCore Archives in a subsequent step. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than the ABCD dump it was created from. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:archArchivingPage.png|border]] In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create a new Archive</small> allows you to create a new archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:archProcess.png|border]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval, or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== a3860547ea5270f5f221675e4f68ea1b24ca1632 405 402 2011-11-15T15:05:00Z JoergHoletschek 3 /* XML Archives */ wikitext text/x-wiki With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: XML Archives simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. DarwinCore Archives (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Archives using the ABCD 2.06 schema (dubbed as “ABCD dumps”) can be converted into one or several DarwinCore Archives in a subsequent step. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than the ABCD dump it was created from. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:archArchivingPage.png|border|600px]] In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create a new Archive</small> allows you to create a new archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:archProcess.png|border|600px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval, or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== 260f2708c4dcdc5f9a9dad1146295af21fcc319a 406 405 2011-11-15T15:05:58Z JoergHoletschek 3 /* XML Archives */ wikitext text/x-wiki With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: XML Archives simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. DarwinCore Archives (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Archives using the ABCD 2.06 schema (dubbed as “ABCD dumps”) can be converted into one or several DarwinCore Archives in a subsequent step. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than the ABCD dump it was created from. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:archArchivingPage.png|border|650px]] In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create a new Archive</small> allows you to create a new archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:archProcess.png|border|650px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval, or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== c38f80e6d8224d2c88da6830b18d3293fe6079ff 407 406 2011-11-15T15:06:29Z JoergHoletschek 3 /* DarwinCore Archives */ wikitext text/x-wiki With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: XML Archives simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. DarwinCore Archives (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Archives using the ABCD 2.06 schema (dubbed as “ABCD dumps”) can be converted into one or several DarwinCore Archives in a subsequent step. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than the ABCD dump it was created from. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:archArchivingPage.png|border|650px]] In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create a new Archive</small> allows you to create a new archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:archProcess.png|border|650px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval, or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== coming soon 14d17b39af0660f5561f779700d349a9521d4863 409 407 2011-11-15T15:11:26Z JoergHoletschek 3 wikitext text/x-wiki With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: XML Archives simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]] DarwinCore Archives (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Archives using the ABCD 2.06 schema (dubbed as “ABCD dumps”) can be converted into one or several DarwinCore Archives in a subsequent step. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than the ABCD dump it was created from. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:archArchivingPage.png|border|650px]] In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create a new Archive</small> allows you to create a new archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:archProcess.png|border|650px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval, or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== coming soon 6fdc4b6968bb2fca6bc96326a38635261cf755a2 410 409 2011-11-15T15:13:58Z JoergHoletschek 3 wikitext text/x-wiki With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: XML Archives simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. DarwinCore Archives (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Archives using the ABCD 2.06 schema (dubbed as “ABCD dumps”) can be converted into one or several DarwinCore Archives in a subsequent step. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than the ABCD dump it was created from. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:archArchivingPage.png|border|650px]] In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create a new Archive</small> allows you to create a new archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:archProcess.png|border|650px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval, or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== coming soon 66ad4a54454358d1e4ecfe6d781e53243a042857 414 410 2011-11-15T15:22:40Z JoergHoletschek 3 /* XML Archives */ wikitext text/x-wiki With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: XML Archives simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. DarwinCore Archives (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Archives using the ABCD 2.06 schema (dubbed as “ABCD dumps”) can be converted into one or several DarwinCore Archives in a subsequent step. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than the ABCD dump it was created from. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:archArchivingPage.png|border]] In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create a new Archive</small> allows you to create a new archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:archProcess.png|border]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval, or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== coming soon bf518a6bc6e6a1b3580cd67f96e47520fc78e7d4 415 414 2011-11-16T09:42:05Z JoergHoletschek 3 wikitext text/x-wiki With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Archives using the ABCD 2.06 schema (dubbed as “ABCD dumps”) can be converted into one or several DarwinCore Archives in a subsequent step. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than the ABCD dump it was created from. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:archArchivingPage.png|border]] In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create a new Archive</small> allows you to create a new archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:archProcess.png|border]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== coming soon 373d67dbd47fbc3e2d75b199db45a47dfa3c9c6a 416 415 2011-11-22T12:03:29Z JoergHoletschek 3 wikitext text/x-wiki With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Archives using the ABCD 2.06 schema (dubbed as “ABCD dumps”) can be converted into one or several DarwinCore Archives in a subsequent step. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than the ABCD dump it was created from. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:archArchivingPage.png|border]] In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create a new Archive</small> allows you to create a new archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:archProcess.png|border]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== coming soon ===Requirements=== ===Installation=== ===Usage=== 10fec79466ea0f527399edd037699fdb02db0867 0 1 395 263 2011-11-15T10:14:52Z JoergHoletschek 3 wikitext text/x-wiki This is the Documentation Wiki of the BioCASe Provider Software. Here you will find tutorials on how to install, configure and debug the BioCASe Provider Software. More detailed information on BioCASe can be found in the [[BeginnersGuide|Beginner's Guide]] and on the [http://www.biocase.org/whats_biocase BioCASe homepage]. [[Image:BioCASeLogoLarge.jpg|link=http://www.biocase.org|150px|right|BioCASe]]Described in one paragraph, the BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. It is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. However, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the BioCASe network (http://www.biocase.org) and the Global Biodiversity Information Facility (http://www.gbif.org). After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. == BioCASe Support == [[Image:mail.png|link=Special:Contact|150px|right|BioCASe Support]] Please get in contact with the BioCASe Support if you * have any questions concerning BioCASe in general or the installation/configuration of the Provider Software, * want to file a bug report, * have any comments, or * want to make a suggestion for a useful new feature in upcoming versions. You can either * use the ''Discussion'' tab of each page to leave us a comment, * use the form on the [[Special:Contact|Contact page]] to send us a message, * or directly mail us to the address given on the [[Special:Contact|Contact page]]. == Table of Contents == ;[[BeginnersGuide|Beginner's Guide]]: What the Provider Software is all about. ;[[Preparation]]: Things to be considered when publishing a database with BioCASe and potential preparatory steps needed. [[Image:support_stamp.png|link=Special:Contact|200px|right|BioCASe Support]] ;[[Installation]]: Tutorial on how to install the BioCASe Provider Software. ;[[DatasourceSetup|Datasource Setup]]: How to set up a new BioCASe web service (aka data source). ;[[ABCD2Mapping|ABCD2 Mapping]]: Tutorial on mapping ABCD2. ;[[CommonABCD2Concepts|Common ABCD2 Concepts]]: List of mandatory, recommended and commonly used ABCD2 elements. ;[[SampleABCDDocument|Sample ABCD2 Document]]: Example ABCD2 document of a comprehensively mapped data provider. ;[[Debugging|Debugging Tutorial]]: Instructions on how to debug a BioCASe web service. ;[[Archiving]]: Learn how to create ABCD Archives or DarwinCore Archives of your published data. ;[[Glossary|Glossary]]: List of terms and abbreviations used in this documentation. ;[[FAQ|FAQ]]: List of frequently asked questions and hopefully helpful answers. ;[[VersionHistory|Version History]]: Change list of current and past versions of the BioCASe Provider Software. 890c3c5d9d3cca0ee3e0881ad28691ec965f7051 8 3 396 230 2011-11-15T10:15:46Z JoergHoletschek 3 wikitext text/x-wiki * Table of Contents ** mainpage|mainpage-description ** BeginnersGuide|Beginner's Guide ** Preparation|Preparation ** Installation|Installation ** DatasourceSetup|Datasource Setup ** ABCD2Mapping|ABCD Mapping ** Debugging|Debugging ** Archiving|Archiving ** Glossary|Glossary ** FAQ|FAQ * ABCD 2.06 ** CommonABCD2Concepts|Common Elements ** SampleABCDDocument|Sample Document * Feedback ** Special:Contact|Contact Us 2b200a709b060898d48ccef9e65ea24f9dc19121 6 81 403 2011-11-15T15:02:34Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 412 403 2011-11-15T15:21:55Z JoergHoletschek 3 uploaded a new version of &quot;[[File:ArchArchivingPage.png]]&quot; wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 82 404 2011-11-15T15:03:16Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 413 404 2011-11-15T15:22:21Z JoergHoletschek 3 uploaded a new version of &quot;[[File:ArchProcess.png]]&quot; wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 83 408 2011-11-15T15:10:56Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 84 411 2011-11-15T15:14:22Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 0 80 417 416 2011-11-22T12:59:09Z JoergHoletschek 3 /* DarwinCore Archives */ wikitext text/x-wiki With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Archives using the ABCD 2.06 schema (dubbed as “ABCD dumps”) can be converted into one or several DarwinCore Archives in a subsequent step. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than the ABCD dump it was created from. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:archArchivingPage.png|border]] In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create a new Archive</small> allows you to create a new archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:archProcess.png|border]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== XML Archives that use the ABCD 2.06 schema can be converted into DarwinCore Archives. Currently, this feature is only available as a standalone command-line tool. Once thoroughly tested, it will be intregrated into the BioCASe Provider Software and can be trigged from the configuration tool. ===Requirements=== Since this tool is currently a standalone application, it doesn't need to be run on the same machine as the BioCASe Provider Software. But of course it can - if you're running BioCASe on a publicly available server and plan to use DwCAs for production, the BioCASe server is definitely the right place. * Java Runtime Environment 1.5 or later installed * 2+G of RAM available * 100M free disk space * One or several ABCD2.06 Archives (created with BPS3+) ===Installation=== # If you don't have a Java Runtime Environment (version 1.5 or later) installed on your machine, get one [http://www.java.com/de/download/ here]. # Download the DwCA Creator from the [http://www.biocase.org/products/provider_software/index.shtml#DwCA_Creator BioCASe Website] and unzip the archive file to a folder on your machine (let's assume it's called ''dwca''). # Test the installation by opening a terminal (''command prompt'' on Windows) and changing to the ''dwca'' folder. Start the tool by running the batch file ''dwca.bat'' on Windows or the shell script ''dwca.sh'' on Linux/MacOS. Since it doesn't do any processing, the test run should take only a few minutes and print some lines similar to following (you can ignore the warning ''No such file or directory'' in the last line): [[File:dwcaTest.png]] ===Usage=== ===Trouble Shooting=== b15a30605610b82e269a476b76b8637e4628dbf8 419 417 2011-11-22T13:02:32Z JoergHoletschek 3 /* DarwinCore Archives */ wikitext text/x-wiki With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Archives using the ABCD 2.06 schema (dubbed as “ABCD dumps”) can be converted into one or several DarwinCore Archives in a subsequent step. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than the ABCD dump it was created from. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:archArchivingPage.png|border]] In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create a new Archive</small> allows you to create a new archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:archProcess.png|border]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== XML Archives that use the ABCD 2.06 schema can be converted into DarwinCore Archives. Currently, this feature is only available as a standalone command-line tool. Once thoroughly tested, it will be intregrated into the BioCASe Provider Software and can be trigged from the configuration tool. ===Requirements=== Since this tool is currently a standalone application, it doesn't need to be run on the same machine as the BioCASe Provider Software. But of course it can - if you're running BioCASe on a publicly available server and plan to use DwCAs for production, the BioCASe server is definitely the right place. * Java Runtime Environment 1.5 or later installed * 2+G of RAM available * 100M free disk space * One or several ABCD2.06 Archives (created with BPS3+) ===Installation=== # If you don't have a Java Runtime Environment (version 1.5 or later) installed on your machine, get one [http://www.java.com/de/download/ here]. # Download the DwCA Creator from the [http://www.biocase.org/products/provider_software/index.shtml#DwCA_Creator BioCASe Website] and unzip the archive file to a folder on your machine (let's assume it's called ''dwca''). # Test the installation by opening a terminal (''command prompt'' on Windows) and changing to the ''dwca'' folder. Start the tool by running the batch file ''dwca.bat'' on Windows or the shell script ''dwca.sh'' on Linux/MacOS. Since it doesn't do any processing, the test run should take only a few seconds and print some lines similar to following (you can ignore the warning ''No such file or directory'' in the last line): [[File:dwcaTest.png]] ===Usage=== ===Trouble Shooting=== 126c3170d3f6889575a51f3703ba8a55593433c6 420 419 2011-11-22T13:24:11Z JoergHoletschek 3 /* DarwinCore Archives */ wikitext text/x-wiki With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Archives using the ABCD 2.06 schema (dubbed as “ABCD dumps”) can be converted into one or several DarwinCore Archives in a subsequent step. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than the ABCD dump it was created from. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:archArchivingPage.png|border]] In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create a new Archive</small> allows you to create a new archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:archProcess.png|border]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== XML Archives that use the ABCD 2.06 schema can be converted into DarwinCore Archives. Currently, this feature is only available as a standalone command-line tool. Once thoroughly tested, it will be intregrated into the BioCASe Provider Software and can be trigged from the configuration tool. ===Requirements=== Since this tool is currently a standalone application, it doesn't need to be run on the same machine as the BioCASe Provider Software. But of course it can - if you're running BioCASe on a publicly available server and plan to use DwCAs for production, the BioCASe server is definitely the right place. * Java Runtime Environment 1.5 or later installed * 2+G of RAM available * 100M free disk space * One or several ABCD2.06 Archives (created with BPS3+) ===Installation=== # If you don't have a Java Runtime Environment (version 1.5 or later) installed on your machine, get one [http://www.java.com/de/download/ here]. # Download the DwCA Creator from the [http://www.biocase.org/products/provider_software/index.shtml#DwCA_Creator BioCASe Website] and unzip the archive file to a folder on your machine (let's assume it's called ''dwca''). # Test the installation by opening a terminal (''command prompt'' on Windows) and changing to the ''dwca'' folder. Start the tool by running the batch file ''dwca.bat'' on Windows or the shell script ''dwca.sh'' on Linux/MacOS. Since it doesn't do any processing, the test run should take only a few seconds and print some lines similar to following (you can ignore the warning ''No such file or directory'' in the last line): [[File:dwcaTest.png]] ===Usage=== The converter can be started with the scripts ''dwca.bat'' (on Windows) and ''dwca.sh'' (on Linux/MacOS) and usually takes one argument - the ABCD archive to be converted. The tool accepts both a file stored in the local file system and an archive available for download at a given URL. Let's assume you've created an ABCD2 dump for your collection. You've placed the archive file (named ''AlgaTerra_ABCD_2.06.zip'') in the same folder as the tool (''root/home/dwca''). To run the conversion on Linux, open a terminal window, change to the ''dwca'' folder and type in the following: <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip</pre> This will unzip the ABCD documents stored in the archive files, create the data files, EML document(s) and the archive descriptor(s) and create the DarwinCore Archives in the folder output: Depending on the size of your collection (number of records), the amount of information published (number of concepts mapped) and the speed of your server (available memory, number of CPUs/cores), this will take from seconds to minutes, for large collections (millions of records) up to hours. ===Trouble Shooting=== 9c88bd34b6a9145a51940959153ad9f359405530 421 420 2011-11-22T13:41:13Z JoergHoletschek 3 /* Usage */ wikitext text/x-wiki With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Archives using the ABCD 2.06 schema (dubbed as “ABCD dumps”) can be converted into one or several DarwinCore Archives in a subsequent step. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than the ABCD dump it was created from. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:archArchivingPage.png|border]] In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create a new Archive</small> allows you to create a new archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:archProcess.png|border]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== XML Archives that use the ABCD 2.06 schema can be converted into DarwinCore Archives. Currently, this feature is only available as a standalone command-line tool. Once thoroughly tested, it will be intregrated into the BioCASe Provider Software and can be trigged from the configuration tool. ===Requirements=== Since this tool is currently a standalone application, it doesn't need to be run on the same machine as the BioCASe Provider Software. But of course it can - if you're running BioCASe on a publicly available server and plan to use DwCAs for production, the BioCASe server is definitely the right place. * Java Runtime Environment 1.5 or later installed * 2+G of RAM available * 100M free disk space * One or several ABCD2.06 Archives (created with BPS3+) ===Installation=== # If you don't have a Java Runtime Environment (version 1.5 or later) installed on your machine, get one [http://www.java.com/de/download/ here]. # Download the DwCA Creator from the [http://www.biocase.org/products/provider_software/index.shtml#DwCA_Creator BioCASe Website] and unzip the archive file to a folder on your machine (let's assume it's called ''dwca''). # Test the installation by opening a terminal (''command prompt'' on Windows) and changing to the ''dwca'' folder. Start the tool by running the batch file ''dwca.bat'' on Windows or the shell script ''dwca.sh'' on Linux/MacOS. Since it doesn't do any processing, the test run should take only a few seconds and print some lines similar to following (you can ignore the warning ''No such file or directory'' in the last line): [[File:dwcaTest.png]] ===Usage=== The converter can be started with the scripts ''dwca.bat'' (on Windows) and ''dwca.sh'' (on Linux/MacOS) and usually takes one argument - the ABCD archive to be converted. The tool accepts both a file stored in the local file system and an archive available for download at a given URL. Let's assume you've created an ABCD2 dump for your collection. You've placed the archive file (named ''AlgaTerra_ABCD_2.06.zip'') in the same folder as the tool (''dwca''). To run the conversion on Linux, open a terminal window, change to the ''dwca'' folder and type in the following: <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip</pre> This will unzip the ABCD documents stored in the archive files, create the data files, EML document(s) and the archive descriptor(s) and create the DarwinCore Archives in the folder output: [[File:dwcaStartLocal.png]] Depending on the size of your collection (number of records), the amount of information published (number of concepts mapped) and the speed of your server (available memory, number of CPUs/cores), this will take from several seconds to some minutes. For large collections (millions of records), it can take up to hours. After each chunk of 50,000 records processed, a line indicating the process will be printed, so you will be able to see if the tool is still running. However, independent of the amount of data to be processed, the last lines of the output will look similar to the following: [[File:dwcaEndLocal.png]] ===Trouble Shooting=== 13622f343925b9fa3fadc2ddca449359b3b39f4b 424 421 2011-11-22T13:46:31Z JoergHoletschek 3 /* Usage */ wikitext text/x-wiki With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Archives using the ABCD 2.06 schema (dubbed as “ABCD dumps”) can be converted into one or several DarwinCore Archives in a subsequent step. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than the ABCD dump it was created from. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:archArchivingPage.png|border]] In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create a new Archive</small> allows you to create a new archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:archProcess.png|border]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== XML Archives that use the ABCD 2.06 schema can be converted into DarwinCore Archives. Currently, this feature is only available as a standalone command-line tool. Once thoroughly tested, it will be intregrated into the BioCASe Provider Software and can be trigged from the configuration tool. ===Requirements=== Since this tool is currently a standalone application, it doesn't need to be run on the same machine as the BioCASe Provider Software. But of course it can - if you're running BioCASe on a publicly available server and plan to use DwCAs for production, the BioCASe server is definitely the right place. * Java Runtime Environment 1.5 or later installed * 2+G of RAM available * 100M free disk space * One or several ABCD2.06 Archives (created with BPS3+) ===Installation=== # If you don't have a Java Runtime Environment (version 1.5 or later) installed on your machine, get one [http://www.java.com/de/download/ here]. # Download the DwCA Creator from the [http://www.biocase.org/products/provider_software/index.shtml#DwCA_Creator BioCASe Website] and unzip the archive file to a folder on your machine (let's assume it's called ''dwca''). # Test the installation by opening a terminal (''command prompt'' on Windows) and changing to the ''dwca'' folder. Start the tool by running the batch file ''dwca.bat'' on Windows or the shell script ''dwca.sh'' on Linux/MacOS. Since it doesn't do any processing, the test run should take only a few seconds and print some lines similar to following (you can ignore the warning ''No such file or directory'' in the last line): [[File:dwcaTest.png]] ===Usage=== The converter can be started with the scripts ''dwca.bat'' (on Windows) and ''dwca.sh'' (on Linux/MacOS) and usually takes one argument - the ABCD archive to be converted. The tool accepts both a file stored in the local file system and an archive available for download at a given URL. Let's assume you've created an ABCD2 dump for your collection. You've placed the archive file (named ''AlgaTerra_ABCD_2.06.zip'') in the same folder as the tool (''dwca''). To run the conversion on Linux, open a terminal window, change to the ''dwca'' folder and type in the following: <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip</pre> This will unzip the ABCD documents stored in the archive file, create the data files, EML document(s) and the archive descriptor(s) and create the DarwinCore Archives in the folder output: [[File:dwcaStartLocal.png]] Depending on the size of your collection (number of records), the amount of information published (number of concepts mapped) and the speed of your server (available memory, number of CPUs/cores), this will take from several seconds to some minutes. For large collections (millions of records), it can take up to hours. After each chunk of 50,000 records processed, a line indicating the progress will be printed, so you will be able to see if the tool is still running. However, independent of the amount of data to be processed, the last lines of the output will look similar to the following: [[File:dwcaEndLocal.png]] ===Trouble Shooting=== 58a835b39a836987452cc604c19ec14e648cf8e1 425 424 2011-11-22T13:54:05Z JoergHoletschek 3 /* Usage */ wikitext text/x-wiki With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Archives using the ABCD 2.06 schema (dubbed as “ABCD dumps”) can be converted into one or several DarwinCore Archives in a subsequent step. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than the ABCD dump it was created from. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:archArchivingPage.png|border]] In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create a new Archive</small> allows you to create a new archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:archProcess.png|border]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== XML Archives that use the ABCD 2.06 schema can be converted into DarwinCore Archives. Currently, this feature is only available as a standalone command-line tool. Once thoroughly tested, it will be intregrated into the BioCASe Provider Software and can be trigged from the configuration tool. ===Requirements=== Since this tool is currently a standalone application, it doesn't need to be run on the same machine as the BioCASe Provider Software. But of course it can - if you're running BioCASe on a publicly available server and plan to use DwCAs for production, the BioCASe server is definitely the right place. * Java Runtime Environment 1.5 or later installed * 2+G of RAM available * 100M free disk space * One or several ABCD2.06 Archives (created with BPS3+) ===Installation=== # If you don't have a Java Runtime Environment (version 1.5 or later) installed on your machine, get one [http://www.java.com/de/download/ here]. # Download the DwCA Creator from the [http://www.biocase.org/products/provider_software/index.shtml#DwCA_Creator BioCASe Website] and unzip the archive file to a folder on your machine (let's assume it's called ''dwca''). # Test the installation by opening a terminal (''command prompt'' on Windows) and changing to the ''dwca'' folder. Start the tool by running the batch file ''dwca.bat'' on Windows or the shell script ''dwca.sh'' on Linux/MacOS. Since it doesn't do any processing, the test run should take only a few seconds and print some lines similar to following (you can ignore the warning ''No such file or directory'' in the last line): [[File:dwcaTest.png]] ===Usage=== The converter can be started with the scripts ''dwca.bat'' (on Windows) and ''dwca.sh'' (on Linux/MacOS) and usually takes one argument - the ABCD archive to be converted. The tool accepts both a file stored in the local file system and an archive available for download at a given URL. Let's assume you've created an ABCD2 dump for your collection. You've placed the archive file (named ''AlgaTerra_ABCD_2.06.zip'') in the same folder as the tool (''dwca''). To run the conversion on Linux, open a terminal window, change to the ''dwca'' folder and type in the following: <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip</pre> This will unzip the ABCD documents stored in the archive file, create the data files, EML document(s) and the archive descriptor(s) and create the DarwinCore Archives in the folder output: [[File:dwcaStartLocal.png]] Depending on the size of your collection (number of records), the amount of information published (number of concepts mapped) and the speed of your server (available memory, number of CPUs/cores), this will take from several seconds to some minutes. For large collections (millions of records), it can take up to hours. After each chunk of 50,000 records processed, a line indicating the progress will be printed, so you will be able to see if the tool is still running. However, independent of the amount of data to be processed, the last lines of the output will look similar to the following: [[File:dwcaEndLocal.png]] The DwCA(s) created can be found in the folder ''output'' created in the base directory (''dwca''). Remember that there will be one DwCA per dataset. So depending upon how many datasets are stored in the source ABCD archive, there will be one or several files in the output folder, each named after the dataset title: [[File:dwcaOutputFolder.png]] ===Trouble Shooting=== 30e4396b64704fd1e91b8836d9a3eee391d3d787 426 425 2011-11-22T14:15:36Z JoergHoletschek 3 /* Usage */ wikitext text/x-wiki With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Archives using the ABCD 2.06 schema (dubbed as “ABCD dumps”) can be converted into one or several DarwinCore Archives in a subsequent step. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than the ABCD dump it was created from. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:archArchivingPage.png|border]] In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create a new Archive</small> allows you to create a new archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:archProcess.png|border]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== XML Archives that use the ABCD 2.06 schema can be converted into DarwinCore Archives. Currently, this feature is only available as a standalone command-line tool. Once thoroughly tested, it will be intregrated into the BioCASe Provider Software and can be trigged from the configuration tool. ===Requirements=== Since this tool is currently a standalone application, it doesn't need to be run on the same machine as the BioCASe Provider Software. But of course it can - if you're running BioCASe on a publicly available server and plan to use DwCAs for production, the BioCASe server is definitely the right place. * Java Runtime Environment 1.5 or later installed * 2+G of RAM available * 100M free disk space * One or several ABCD2.06 Archives (created with BPS3+) ===Installation=== # If you don't have a Java Runtime Environment (version 1.5 or later) installed on your machine, get one [http://www.java.com/de/download/ here]. # Download the DwCA Creator from the [http://www.biocase.org/products/provider_software/index.shtml#DwCA_Creator BioCASe Website] and unzip the archive file to a folder on your machine (let's assume it's called ''dwca''). # Test the installation by opening a terminal (''command prompt'' on Windows) and changing to the ''dwca'' folder. Start the tool by running the batch file ''dwca.bat'' on Windows or the shell script ''dwca.sh'' on Linux/MacOS. Since it doesn't do any processing, the test run should take only a few seconds and print some lines similar to following (you can ignore the warning ''No such file or directory'' in the last line): [[File:dwcaTest.png]] ===Usage=== The converter can be started with the scripts ''dwca.bat'' (on Windows) and ''dwca.sh'' (on Linux/MacOS) and usually takes one argument - the ABCD archive to be converted. The tool accepts both a file stored in the local file system and an archive available for download at a given URL. Let's assume you've created an ABCD2 dump for your collection. You've placed the archive file (named ''AlgaTerra_ABCD_2.06.zip'') in the same folder as the tool (''dwca''). To run the conversion on Linux, open a terminal window, change to the ''dwca'' folder and type in the following: <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command: <pre>dwca.bat AlgaTerra_ABCD_2.06.zip</pre> This will unzip the ABCD documents stored in the archive file, create the data files, EML document(s) and the archive descriptor(s): [[File:dwcaStartLocal.png]] Depending on the size of your collection (number of records), the amount of information published (number of concepts mapped) and the speed of your server (available memory, number of CPUs/cores), this will take from several seconds to some minutes. For large collections (millions of records), it can take up to hours. After each chunk of 50,000 records processed, a line indicating the progress will be printed, so you will be able to see if the tool is still running. However, independent of the amount of data to be processed, the last lines of the output will look similar to the following: [[File:dwcaEndLocal.png]] The DwCA(s) created can be found in the folder ''output'' created in the base directory (''dwca''). Remember that there will be one DwCA per dataset. So depending upon how many datasets are stored in the source ABCD archive, there will be one or several files in the output folder. On Linux and MacOS, the shell script ''dwca.sh'' will zip all files that make up one DarwinCore Archive into an archive file named after the dataset title. So you could send this files directly to a potential DwCA consumer: [[File:dwcaOutputFolderLinux.png]] On Windows, the batch file ''dwca.bat'' of the prototype doesn't do that zipping. So if you go to the output file after the conversion has finished, you will see several text files and XML documents, all named after the pattern ''<dataset>_<filename>''. So for the example above, the output folder would look like this: In order to create a DwCA that can be sent to a consumer, please do the follwing for each ''<dataset>'': # Note down the dataset title, # Remove the dataset title in the file names (''<dataset>_<filename>'' becomes ''dataset_title''), # Create a zip named ''<dataset>.zip'' with all the files you just renamed. This lack of functionality is due to the limited capabilities of Windows batch files. Once integrated into the Provider Software, you won't need to zip up files manually. ===Trouble Shooting=== 1982ef72985a64b896c0f5975c0c09527304330d 427 426 2011-11-22T14:24:12Z JoergHoletschek 3 /* Usage */ wikitext text/x-wiki With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Archives using the ABCD 2.06 schema (dubbed as “ABCD dumps”) can be converted into one or several DarwinCore Archives in a subsequent step. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than the ABCD dump it was created from. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:archArchivingPage.png|border]] In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create a new Archive</small> allows you to create a new archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:archProcess.png|border]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== XML Archives that use the ABCD 2.06 schema can be converted into DarwinCore Archives. Currently, this feature is only available as a standalone command-line tool. Once thoroughly tested, it will be intregrated into the BioCASe Provider Software and can be trigged from the configuration tool. ===Requirements=== Since this tool is currently a standalone application, it doesn't need to be run on the same machine as the BioCASe Provider Software. But of course it can - if you're running BioCASe on a publicly available server and plan to use DwCAs for production, the BioCASe server is definitely the right place. * Java Runtime Environment 1.5 or later installed * 2+G of RAM available * 100M free disk space * One or several ABCD2.06 Archives (created with BPS3+) ===Installation=== # If you don't have a Java Runtime Environment (version 1.5 or later) installed on your machine, get one [http://www.java.com/de/download/ here]. # Download the DwCA Creator from the [http://www.biocase.org/products/provider_software/index.shtml#DwCA_Creator BioCASe Website] and unzip the archive file to a folder on your machine (let's assume it's called ''dwca''). # Test the installation by opening a terminal (''command prompt'' on Windows) and changing to the ''dwca'' folder. Start the tool by running the batch file ''dwca.bat'' on Windows or the shell script ''dwca.sh'' on Linux/MacOS. Since it doesn't do any processing, the test run should take only a few seconds and print some lines similar to following (you can ignore the warning ''No such file or directory'' in the last line): [[File:dwcaTest.png]] ===Usage=== The converter can be started with the scripts ''dwca.bat'' (on Windows) and ''dwca.sh'' (on Linux/MacOS) and usually takes one argument - the ABCD archive to be converted. The tool accepts both a file stored in the local file system and an archive available for download at a given URL. Let's assume you've created an ABCD2 dump for your collection. You've placed the archive file (named ''AlgaTerra_ABCD_2.06.zip'') in the same folder as the tool (''dwca''). To run the conversion on Linux, open a terminal window, change to the ''dwca'' folder and type in the following: <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command: <pre>dwca.bat AlgaTerra_ABCD_2.06.zip</pre> This will unzip the ABCD documents stored in the archive file, create the data files, EML document(s) and the archive descriptor(s): [[File:dwcaStartLocal.png]] Depending on the size of your collection (number of records), the amount of information published (number of concepts mapped) and the speed of your server (available memory, number of CPUs/cores), this will take from several seconds to some minutes. For large collections (millions of records), it can take up to hours. After each chunk of 50,000 records processed, a line indicating the progress will be printed, so you will be able to see if the tool is still running. However, independent of the amount of data to be processed, the last lines of the output will look similar to the following: [[File:dwcaEndLocal.png]] The DwCA(s) created can be found in the folder ''output'' created in the base directory (''dwca''). Remember that there will be one DwCA per dataset. So depending upon how many datasets are stored in the source ABCD archive, there will be one or several files in the output folder. On Linux and MacOS, the shell script ''dwca.sh'' will zip all files that make up one DarwinCore Archive into an archive file named after the dataset title. So you could send this files directly to a potential DwCA consumer: [[File:dwcaOutputFolderLinux.png]] On Windows, the batch file ''dwca.bat'' of the prototype doesn't do that zipping. So if you go to the output file after the conversion has finished, you will see several text files and XML documents, all named after the pattern ''<dataset> <filename>''. So for the example above, the output folder would look like this: [[File:dwcaOutputFolderWindows.png]] In order to create a DwCA that can be sent to a consumer, please do the follwing for each ''<dataset>'': # Note down the dataset title, # Remove the dataset title in the file names (''<dataset> <filename>'' becomes ''<filename>''), # Create a zip named ''<dataset>.zip'' with all the files you just renamed. This lack of functionality is due to the limited capabilities of Windows batch files. Once integrated into the Provider Software, you won't need to zip up files manually. ===Trouble Shooting=== a0211b27c01099922e40e1065010f3464bc254f4 430 427 2011-11-22T14:27:30Z JoergHoletschek 3 wikitext text/x-wiki With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Archives using the ABCD 2.06 schema (dubbed as “ABCD dumps”) can be converted into one or several DarwinCore Archives in a subsequent step. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than the ABCD dump it was created from. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:archArchivingPage.png|border]] In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create a new Archive</small> allows you to create a new archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:archProcess.png|border]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== XML Archives that use the ABCD 2.06 schema can be converted into DarwinCore Archives. Currently, this feature is only available as a standalone command-line tool. Once thoroughly tested, it will be intregrated into the BioCASe Provider Software and can be trigged from the configuration tool. ===Requirements=== Since this tool is currently a standalone application, it doesn't need to be run on the same machine as the BioCASe Provider Software. But of course it can - if you're running BioCASe on a publicly available server and plan to use DwCAs for production, the BioCASe server is definitely the right place. * Java Runtime Environment 1.5 or later installed * 2+G of RAM available * 100M free disk space * One or several ABCD2.06 Archives (created with BPS3+) ===Installation=== # If you don't have a Java Runtime Environment (version 1.5 or later) installed on your machine, get one [http://www.java.com/de/download/ here]. # Download the DwCA Creator from the [http://www.biocase.org/products/provider_software/index.shtml#DwCA_Creator BioCASe Website] and unzip the archive file to a folder on your machine (let's assume it's called ''dwca''). # Test the installation by opening a terminal (''command prompt'' on Windows) and changing to the ''dwca'' folder. Start the tool by running the batch file ''dwca.bat'' on Windows or the shell script ''dwca.sh'' on Linux/MacOS. Since it doesn't do any processing, the test run should take only a few seconds and print some lines similar to following (you can ignore the warning ''No such file or directory'' in the last line): [[File:dwcaTest.png]] ===Usage=== The converter can be started with the scripts ''dwca.bat'' (on Windows) and ''dwca.sh'' (on Linux/MacOS) and usually takes one argument - the ABCD archive to be converted. The tool accepts both a file stored in the local file system and an archive available for download at a given URL. Let's assume you've created an ABCD2 dump for your collection. You've placed the archive file (named ''AlgaTerra_ABCD_2.06.zip'') in the same folder as the tool (''dwca''). To run the conversion on Linux, open a terminal window, change to the ''dwca'' folder and type in the following: <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command: <pre>dwca.bat AlgaTerra_ABCD_2.06.zip</pre> This will unzip the ABCD documents stored in the archive file, create the data files, EML document(s) and the archive descriptor(s): [[File:dwcaStartLocal.png]] Depending on the size of your collection (number of records), the amount of information published (number of concepts mapped) and the speed of your server (available memory, number of CPUs/cores), this will take from several seconds to some minutes. For large collections (millions of records), it can take up to hours. After each chunk of 50,000 records processed, a line indicating the progress will be printed, so you will be able to see if the tool is still running. However, independent of the amount of data to be processed, the last lines of the output will look similar to the following: [[File:dwcaEndLocal.png]] ===Output=== The DwCA(s) created can be found in the folder ''output'' created in the base directory (''dwca''). Remember that there will be one DwCA per dataset. So depending upon how many datasets are stored in the source ABCD archive, there will be one or several files in the output folder. On Linux and MacOS, the shell script ''dwca.sh'' will zip all files that make up one DarwinCore Archive into an archive file named after the dataset title. So you could send this files directly to a potential DwCA consumer: [[File:dwcaOutputFolderLinux.png]] On Windows, the batch file ''dwca.bat'' of the prototype doesn't do that zipping. So if you go to the output file after the conversion has finished, you will see several text files and XML documents, all named after the pattern ''<dataset> <filename>''. So for the example above, the output folder would look like this: [[File:dwcaOutputFolderWindows.png]] In order to create a DwCA that can be sent to a consumer, please do the follwing for each ''<dataset>'': # Note down the dataset title, # Remove the dataset title in the file names (''<dataset> <filename>'' becomes ''<filename>''), # Create a zip named ''<dataset>.zip'' with all the files you just renamed. This lack of functionality is due to the limited capabilities of Windows batch files. Once integrated into the Provider Software, you won't need to zip up files manually. ===Trouble Shooting=== 0373dee516a95da7835b764e48a47e7b16c85e28 431 430 2011-11-22T15:04:44Z JoergHoletschek 3 /* Usage */ wikitext text/x-wiki With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Archives using the ABCD 2.06 schema (dubbed as “ABCD dumps”) can be converted into one or several DarwinCore Archives in a subsequent step. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than the ABCD dump it was created from. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:archArchivingPage.png|border]] In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create a new Archive</small> allows you to create a new archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:archProcess.png|border]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== XML Archives that use the ABCD 2.06 schema can be converted into DarwinCore Archives. Currently, this feature is only available as a standalone command-line tool. Once thoroughly tested, it will be intregrated into the BioCASe Provider Software and can be trigged from the configuration tool. ===Requirements=== Since this tool is currently a standalone application, it doesn't need to be run on the same machine as the BioCASe Provider Software. But of course it can - if you're running BioCASe on a publicly available server and plan to use DwCAs for production, the BioCASe server is definitely the right place. * Java Runtime Environment 1.5 or later installed * 2+G of RAM available * 100M free disk space * One or several ABCD2.06 Archives (created with BPS3+) ===Installation=== # If you don't have a Java Runtime Environment (version 1.5 or later) installed on your machine, get one [http://www.java.com/de/download/ here]. # Download the DwCA Creator from the [http://www.biocase.org/products/provider_software/index.shtml#DwCA_Creator BioCASe Website] and unzip the archive file to a folder on your machine (let's assume it's called ''dwca''). # Test the installation by opening a terminal (''command prompt'' on Windows) and changing to the ''dwca'' folder. Start the tool by running the batch file ''dwca.bat'' on Windows or the shell script ''dwca.sh'' on Linux/MacOS. Since it doesn't do any processing, the test run should take only a few seconds and print some lines similar to following (you can ignore the warning ''No such file or directory'' in the last line): [[File:dwcaTest.png]] ===Usage=== The converter can be started with the scripts ''dwca.bat'' (on Windows) and ''dwca.sh'' (on Linux/MacOS) and usually takes one argument - the ABCD archive to be converted. The tool accepts both a file stored in the local file system and an archive available for download at a given URL. Let's assume you've created an ABCD2 dump for your collection. You've placed the archive file (named ''AlgaTerra_ABCD_2.06.zip'') in the same folder as the tool (''dwca''). To run the conversion on Linux, open a terminal window, change to the ''dwca'' folder and type in the following: <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command. Note that the base folder for the transformer is the ''kettle'' folder, so you need to specify ''..\'' to direct the transformer to its parent folder for finding the archive file (alternatively, you can put the archive file in the ''kettle'' subfolder and thus spare the ''..\''): <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip</pre> This will unzip the ABCD documents stored in the archive file, create the data files, EML document(s) and the archive descriptor(s): [[File:dwcaStartLocal.png]] Depending on the size of your collection (number of records), the amount of information published (number of concepts mapped) and the speed of your server (available memory, number of CPUs/cores), this will take from several seconds to some minutes. For large collections (millions of records), it can take up to hours. After each chunk of 50,000 records processed, a line indicating the progress will be printed, so you will be able to see if the tool is still running. However, independent of the amount of data to be processed, the last lines of the output will look similar to the following: [[File:dwcaEndLocal.png]] When creating an ABCD Archive with the BioCASe Provider Software, it becomes automatically available for download on the BioCASe installation after the process has finished. The DwCA Creator can download the archive directly from this location. So if you're BioCASe installation is accessible from the machine you're running the prototype on, you can spare the trouble of moving the file manually and provide the URL instead of a local file name. To get the download URL for an ABCD Archive you've created, find the archive in question in the list of existing archives in the configuration tool (see [[Archiving#XML_Archives|here]] for more on that). Right-click on the Download link and choose ''Copy link''. Then open a terminal window for the creator tool and provide the link intead of a file name: <pre>./dwca.sh http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command: <pre>dwca.bat http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> ===Output=== The DwCA(s) created can be found in the folder ''output'' created in the base directory (''dwca''). Remember that there will be one DwCA per dataset. So depending upon how many datasets are stored in the source ABCD archive, there will be one or several files in the output folder. On Linux and MacOS, the shell script ''dwca.sh'' will zip all files that make up one DarwinCore Archive into an archive file named after the dataset title. So you could send this files directly to a potential DwCA consumer: [[File:dwcaOutputFolderLinux.png]] On Windows, the batch file ''dwca.bat'' of the prototype doesn't do that zipping. So if you go to the output file after the conversion has finished, you will see several text files and XML documents, all named after the pattern ''<dataset> <filename>''. So for the example above, the output folder would look like this: [[File:dwcaOutputFolderWindows.png]] In order to create a DwCA that can be sent to a consumer, please do the follwing for each ''<dataset>'': # Note down the dataset title, # Remove the dataset title in the file names (''<dataset> <filename>'' becomes ''<filename>''), # Create a zip named ''<dataset>.zip'' with all the files you just renamed. This lack of functionality is due to the limited capabilities of Windows batch files. Once integrated into the Provider Software, you won't need to zip up files manually. ===Trouble Shooting=== dde3357351b6dae1d25d260aaf4b2441938b21af 432 431 2011-11-22T15:06:59Z JoergHoletschek 3 /* XML Archives */ wikitext text/x-wiki With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Archives using the ABCD 2.06 schema (dubbed as “ABCD dumps”) can be converted into one or several DarwinCore Archives in a subsequent step. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than the ABCD dump it was created from. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:archArchivingPage.png|border]] ===Usage=== In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create a new Archive</small> allows you to create a new archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:archProcess.png|border]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== XML Archives that use the ABCD 2.06 schema can be converted into DarwinCore Archives. Currently, this feature is only available as a standalone command-line tool. Once thoroughly tested, it will be intregrated into the BioCASe Provider Software and can be trigged from the configuration tool. ===Requirements=== Since this tool is currently a standalone application, it doesn't need to be run on the same machine as the BioCASe Provider Software. But of course it can - if you're running BioCASe on a publicly available server and plan to use DwCAs for production, the BioCASe server is definitely the right place. * Java Runtime Environment 1.5 or later installed * 2+G of RAM available * 100M free disk space * One or several ABCD2.06 Archives (created with BPS3+) ===Installation=== # If you don't have a Java Runtime Environment (version 1.5 or later) installed on your machine, get one [http://www.java.com/de/download/ here]. # Download the DwCA Creator from the [http://www.biocase.org/products/provider_software/index.shtml#DwCA_Creator BioCASe Website] and unzip the archive file to a folder on your machine (let's assume it's called ''dwca''). # Test the installation by opening a terminal (''command prompt'' on Windows) and changing to the ''dwca'' folder. Start the tool by running the batch file ''dwca.bat'' on Windows or the shell script ''dwca.sh'' on Linux/MacOS. Since it doesn't do any processing, the test run should take only a few seconds and print some lines similar to following (you can ignore the warning ''No such file or directory'' in the last line): [[File:dwcaTest.png]] ===Usage=== The converter can be started with the scripts ''dwca.bat'' (on Windows) and ''dwca.sh'' (on Linux/MacOS) and usually takes one argument - the ABCD archive to be converted. The tool accepts both a file stored in the local file system and an archive available for download at a given URL. Let's assume you've created an ABCD2 dump for your collection. You've placed the archive file (named ''AlgaTerra_ABCD_2.06.zip'') in the same folder as the tool (''dwca''). To run the conversion on Linux, open a terminal window, change to the ''dwca'' folder and type in the following: <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command. Note that the base folder for the transformer is the ''kettle'' folder, so you need to specify ''..\'' to direct the transformer to its parent folder for finding the archive file (alternatively, you can put the archive file in the ''kettle'' subfolder and thus spare the ''..\''): <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip</pre> This will unzip the ABCD documents stored in the archive file, create the data files, EML document(s) and the archive descriptor(s): [[File:dwcaStartLocal.png]] Depending on the size of your collection (number of records), the amount of information published (number of concepts mapped) and the speed of your server (available memory, number of CPUs/cores), this will take from several seconds to some minutes. For large collections (millions of records), it can take up to hours. After each chunk of 50,000 records processed, a line indicating the progress will be printed, so you will be able to see if the tool is still running. However, independent of the amount of data to be processed, the last lines of the output will look similar to the following: [[File:dwcaEndLocal.png]] When creating an ABCD Archive with the BioCASe Provider Software, it becomes automatically available for download on the BioCASe installation after the process has finished. The DwCA Creator can download the archive directly from this location. So if you're BioCASe installation is accessible from the machine you're running the prototype on, you can spare the trouble of moving the file manually and provide the URL instead of a local file name. To get the download URL for an ABCD Archive you've created, find the archive in question in the list of existing archives in the configuration tool (see [[Archiving#XML_Archives|here]] for more on that). Right-click on the Download link and choose ''Copy link''. Then open a terminal window for the creator tool and provide the link intead of a file name: <pre>./dwca.sh http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command: <pre>dwca.bat http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> ===Output=== The DwCA(s) created can be found in the folder ''output'' created in the base directory (''dwca''). Remember that there will be one DwCA per dataset. So depending upon how many datasets are stored in the source ABCD archive, there will be one or several files in the output folder. On Linux and MacOS, the shell script ''dwca.sh'' will zip all files that make up one DarwinCore Archive into an archive file named after the dataset title. So you could send this files directly to a potential DwCA consumer: [[File:dwcaOutputFolderLinux.png]] On Windows, the batch file ''dwca.bat'' of the prototype doesn't do that zipping. So if you go to the output file after the conversion has finished, you will see several text files and XML documents, all named after the pattern ''<dataset> <filename>''. So for the example above, the output folder would look like this: [[File:dwcaOutputFolderWindows.png]] In order to create a DwCA that can be sent to a consumer, please do the follwing for each ''<dataset>'': # Note down the dataset title, # Remove the dataset title in the file names (''<dataset> <filename>'' becomes ''<filename>''), # Create a zip named ''<dataset>.zip'' with all the files you just renamed. This lack of functionality is due to the limited capabilities of Windows batch files. Once integrated into the Provider Software, you won't need to zip up files manually. ===Trouble Shooting=== 5afb37a57ccc3695e8e90c020f06a094ba61dcf9 433 432 2011-11-22T15:29:53Z JoergHoletschek 3 /* Installation */ wikitext text/x-wiki With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Archives using the ABCD 2.06 schema (dubbed as “ABCD dumps”) can be converted into one or several DarwinCore Archives in a subsequent step. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than the ABCD dump it was created from. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:archArchivingPage.png|border]] ===Usage=== In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create a new Archive</small> allows you to create a new archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:archProcess.png|border]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== XML Archives that use the ABCD 2.06 schema can be converted into DarwinCore Archives. Currently, this feature is only available as a standalone command-line tool. Once thoroughly tested, it will be intregrated into the BioCASe Provider Software and can be trigged from the configuration tool. ===Requirements=== Since this tool is currently a standalone application, it doesn't need to be run on the same machine as the BioCASe Provider Software. But of course it can - if you're running BioCASe on a publicly available server and plan to use DwCAs for production, the BioCASe server is definitely the right place. * Java Runtime Environment 1.5 or later installed * 2+G of RAM available * 100M free disk space * One or several ABCD2.06 Archives (created with BPS3+) ===Installation=== # If you don't have a Java Runtime Environment (version 1.5 or later) installed on your machine, get one [http://www.java.com/de/download/ here]. # Download the DwCA Creator from the [http://www.biocase.org/products/provider_software/index.shtml#DwCA_Creator BioCASe Website] and unzip the archive file to your machine. This will create a folder ''dwca'' for the tool. # Test the installation by opening a terminal (''command prompt'' on Windows) and changing to the ''dwca'' folder. Start the tool by running the batch file ''dwca.bat'' on Windows or the shell script ''dwca.sh'' on Linux/MacOS. Since it doesn't do any processing, the test run should take only a few seconds and print some lines similar to following (you can ignore the warning ''No such file or directory'' in the last line): [[File:dwcaTest.png]] ===Usage=== The converter can be started with the scripts ''dwca.bat'' (on Windows) and ''dwca.sh'' (on Linux/MacOS) and usually takes one argument - the ABCD archive to be converted. The tool accepts both a file stored in the local file system and an archive available for download at a given URL. Let's assume you've created an ABCD2 dump for your collection. You've placed the archive file (named ''AlgaTerra_ABCD_2.06.zip'') in the same folder as the tool (''dwca''). To run the conversion on Linux, open a terminal window, change to the ''dwca'' folder and type in the following: <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command. Note that the base folder for the transformer is the ''kettle'' folder, so you need to specify ''..\'' to direct the transformer to its parent folder for finding the archive file (alternatively, you can put the archive file in the ''kettle'' subfolder and thus spare the ''..\''): <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip</pre> This will unzip the ABCD documents stored in the archive file, create the data files, EML document(s) and the archive descriptor(s): [[File:dwcaStartLocal.png]] Depending on the size of your collection (number of records), the amount of information published (number of concepts mapped) and the speed of your server (available memory, number of CPUs/cores), this will take from several seconds to some minutes. For large collections (millions of records), it can take up to hours. After each chunk of 50,000 records processed, a line indicating the progress will be printed, so you will be able to see if the tool is still running. However, independent of the amount of data to be processed, the last lines of the output will look similar to the following: [[File:dwcaEndLocal.png]] When creating an ABCD Archive with the BioCASe Provider Software, it becomes automatically available for download on the BioCASe installation after the process has finished. The DwCA Creator can download the archive directly from this location. So if you're BioCASe installation is accessible from the machine you're running the prototype on, you can spare the trouble of moving the file manually and provide the URL instead of a local file name. To get the download URL for an ABCD Archive you've created, find the archive in question in the list of existing archives in the configuration tool (see [[Archiving#XML_Archives|here]] for more on that). Right-click on the Download link and choose ''Copy link''. Then open a terminal window for the creator tool and provide the link intead of a file name: <pre>./dwca.sh http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command: <pre>dwca.bat http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> ===Output=== The DwCA(s) created can be found in the folder ''output'' created in the base directory (''dwca''). Remember that there will be one DwCA per dataset. So depending upon how many datasets are stored in the source ABCD archive, there will be one or several files in the output folder. On Linux and MacOS, the shell script ''dwca.sh'' will zip all files that make up one DarwinCore Archive into an archive file named after the dataset title. So you could send this files directly to a potential DwCA consumer: [[File:dwcaOutputFolderLinux.png]] On Windows, the batch file ''dwca.bat'' of the prototype doesn't do that zipping. So if you go to the output file after the conversion has finished, you will see several text files and XML documents, all named after the pattern ''<dataset> <filename>''. So for the example above, the output folder would look like this: [[File:dwcaOutputFolderWindows.png]] In order to create a DwCA that can be sent to a consumer, please do the follwing for each ''<dataset>'': # Note down the dataset title, # Remove the dataset title in the file names (''<dataset> <filename>'' becomes ''<filename>''), # Create a zip named ''<dataset>.zip'' with all the files you just renamed. This lack of functionality is due to the limited capabilities of Windows batch files. Once integrated into the Provider Software, you won't need to zip up files manually. ===Trouble Shooting=== 640970b0ce26a61c1376ebe7246859d66cabf168 434 433 2011-11-22T15:48:47Z JoergHoletschek 3 /* Trouble Shooting */ wikitext text/x-wiki With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Archives using the ABCD 2.06 schema (dubbed as “ABCD dumps”) can be converted into one or several DarwinCore Archives in a subsequent step. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than the ABCD dump it was created from. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:archArchivingPage.png|border]] ===Usage=== In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create a new Archive</small> allows you to create a new archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:archProcess.png|border]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== XML Archives that use the ABCD 2.06 schema can be converted into DarwinCore Archives. Currently, this feature is only available as a standalone command-line tool. Once thoroughly tested, it will be intregrated into the BioCASe Provider Software and can be trigged from the configuration tool. ===Requirements=== Since this tool is currently a standalone application, it doesn't need to be run on the same machine as the BioCASe Provider Software. But of course it can - if you're running BioCASe on a publicly available server and plan to use DwCAs for production, the BioCASe server is definitely the right place. * Java Runtime Environment 1.5 or later installed * 2+G of RAM available * 100M free disk space * One or several ABCD2.06 Archives (created with BPS3+) ===Installation=== # If you don't have a Java Runtime Environment (version 1.5 or later) installed on your machine, get one [http://www.java.com/de/download/ here]. # Download the DwCA Creator from the [http://www.biocase.org/products/provider_software/index.shtml#DwCA_Creator BioCASe Website] and unzip the archive file to your machine. This will create a folder ''dwca'' for the tool. # Test the installation by opening a terminal (''command prompt'' on Windows) and changing to the ''dwca'' folder. Start the tool by running the batch file ''dwca.bat'' on Windows or the shell script ''dwca.sh'' on Linux/MacOS. Since it doesn't do any processing, the test run should take only a few seconds and print some lines similar to following (you can ignore the warning ''No such file or directory'' in the last line): [[File:dwcaTest.png]] ===Usage=== The converter can be started with the scripts ''dwca.bat'' (on Windows) and ''dwca.sh'' (on Linux/MacOS) and usually takes one argument - the ABCD archive to be converted. The tool accepts both a file stored in the local file system and an archive available for download at a given URL. Let's assume you've created an ABCD2 dump for your collection. You've placed the archive file (named ''AlgaTerra_ABCD_2.06.zip'') in the same folder as the tool (''dwca''). To run the conversion on Linux, open a terminal window, change to the ''dwca'' folder and type in the following: <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command. Note that the base folder for the transformer is the ''kettle'' folder, so you need to specify ''..\'' to direct the transformer to its parent folder for finding the archive file (alternatively, you can put the archive file in the ''kettle'' subfolder and thus spare the ''..\''): <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip</pre> This will unzip the ABCD documents stored in the archive file, create the data files, EML document(s) and the archive descriptor(s): [[File:dwcaStartLocal.png]] Depending on the size of your collection (number of records), the amount of information published (number of concepts mapped) and the speed of your server (available memory, number of CPUs/cores), this will take from several seconds to some minutes. For large collections (millions of records), it can take up to hours. After each chunk of 50,000 records processed, a line indicating the progress will be printed, so you will be able to see if the tool is still running. However, independent of the amount of data to be processed, the last lines of the output will look similar to the following: [[File:dwcaEndLocal.png]] When creating an ABCD Archive with the BioCASe Provider Software, it becomes automatically available for download on the BioCASe installation after the process has finished. The DwCA Creator can download the archive directly from this location. So if you're BioCASe installation is accessible from the machine you're running the prototype on, you can spare the trouble of moving the file manually and provide the URL instead of a local file name. To get the download URL for an ABCD Archive you've created, find the archive in question in the list of existing archives in the configuration tool (see [[Archiving#XML_Archives|here]] for more on that). Right-click on the Download link and choose ''Copy link''. Then open a terminal window for the creator tool and provide the link intead of a file name: <pre>./dwca.sh http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command: <pre>dwca.bat http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> ===Output=== The DwCA(s) created can be found in the folder ''output'' created in the base directory (''dwca''). Remember that there will be one DwCA per dataset. So depending upon how many datasets are stored in the source ABCD archive, there will be one or several files in the output folder. On Linux and MacOS, the shell script ''dwca.sh'' will zip all files that make up one DarwinCore Archive into an archive file named after the dataset title. So you could send this files directly to a potential DwCA consumer: [[File:dwcaOutputFolderLinux.png]] On Windows, the batch file ''dwca.bat'' of the prototype doesn't do that zipping. So if you go to the output file after the conversion has finished, you will see several text files and XML documents, all named after the pattern ''<dataset> <filename>''. So for the example above, the output folder would look like this: [[File:dwcaOutputFolderWindows.png]] In order to create a DwCA that can be sent to a consumer, please do the follwing for each ''<dataset>'': # Note down the dataset title, # Remove the dataset title in the file names (''<dataset> <filename>'' becomes ''<filename>''), # Create a zip named ''<dataset>.zip'' with all the files you just renamed. This lack of functionality is due to the limited capabilities of Windows batch files. Once integrated into the Provider Software, you won't need to zip up files manually. ===Trouble Shooting=== The default settings should be working for small and medium-sized collections (up to 500,000 records). If the processing is cancelled with an OutOfMemoryError, you should try the follwing: '''Increasing the maximum memory used by the transformer''' Per default, the transformer will use up to 1GB of RAM for the transformation process. In case of an OutOfMemoryError, you should increase this limit to 2GB. Don't be stingy with memory, since it will be only used during the tranformation process and be freed afterwards. Moreover, this is only the upper limit, so actual memory usage might well stay below this limit. To allow the transformer to use more memory, go to the folder ''kettle'' of your installation. Open the file ''kitchen.sh'' (Linux/MacOS) or ''kitchen.bat'' (Windows) in a text editor. Use the ''Find'' feature of the editor to go to the spot in file that specifies the memory limit. Replace the value ''1024'' with the amount of memory you want to allow the transformation to use, so e.g. ''2048'' for 2GB: [[File:dwcaSettingMemory.png]] Save the file and restart the transformation. '''Lowering the number of records''' efc96a5a9c0b9b8c74886dc6f28cf83613870203 436 434 2011-11-22T15:50:53Z JoergHoletschek 3 /* Trouble Shooting */ wikitext text/x-wiki With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Archives using the ABCD 2.06 schema (dubbed as “ABCD dumps”) can be converted into one or several DarwinCore Archives in a subsequent step. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than the ABCD dump it was created from. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:archArchivingPage.png|border]] ===Usage=== In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create a new Archive</small> allows you to create a new archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:archProcess.png|border]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== XML Archives that use the ABCD 2.06 schema can be converted into DarwinCore Archives. Currently, this feature is only available as a standalone command-line tool. Once thoroughly tested, it will be intregrated into the BioCASe Provider Software and can be trigged from the configuration tool. ===Requirements=== Since this tool is currently a standalone application, it doesn't need to be run on the same machine as the BioCASe Provider Software. But of course it can - if you're running BioCASe on a publicly available server and plan to use DwCAs for production, the BioCASe server is definitely the right place. * Java Runtime Environment 1.5 or later installed * 2+G of RAM available * 100M free disk space * One or several ABCD2.06 Archives (created with BPS3+) ===Installation=== # If you don't have a Java Runtime Environment (version 1.5 or later) installed on your machine, get one [http://www.java.com/de/download/ here]. # Download the DwCA Creator from the [http://www.biocase.org/products/provider_software/index.shtml#DwCA_Creator BioCASe Website] and unzip the archive file to your machine. This will create a folder ''dwca'' for the tool. # Test the installation by opening a terminal (''command prompt'' on Windows) and changing to the ''dwca'' folder. Start the tool by running the batch file ''dwca.bat'' on Windows or the shell script ''dwca.sh'' on Linux/MacOS. Since it doesn't do any processing, the test run should take only a few seconds and print some lines similar to following (you can ignore the warning ''No such file or directory'' in the last line): [[File:dwcaTest.png]] ===Usage=== The converter can be started with the scripts ''dwca.bat'' (on Windows) and ''dwca.sh'' (on Linux/MacOS) and usually takes one argument - the ABCD archive to be converted. The tool accepts both a file stored in the local file system and an archive available for download at a given URL. Let's assume you've created an ABCD2 dump for your collection. You've placed the archive file (named ''AlgaTerra_ABCD_2.06.zip'') in the same folder as the tool (''dwca''). To run the conversion on Linux, open a terminal window, change to the ''dwca'' folder and type in the following: <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command. Note that the base folder for the transformer is the ''kettle'' folder, so you need to specify ''..\'' to direct the transformer to its parent folder for finding the archive file (alternatively, you can put the archive file in the ''kettle'' subfolder and thus spare the ''..\''): <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip</pre> This will unzip the ABCD documents stored in the archive file, create the data files, EML document(s) and the archive descriptor(s): [[File:dwcaStartLocal.png]] Depending on the size of your collection (number of records), the amount of information published (number of concepts mapped) and the speed of your server (available memory, number of CPUs/cores), this will take from several seconds to some minutes. For large collections (millions of records), it can take up to hours. After each chunk of 50,000 records processed, a line indicating the progress will be printed, so you will be able to see if the tool is still running. However, independent of the amount of data to be processed, the last lines of the output will look similar to the following: [[File:dwcaEndLocal.png]] When creating an ABCD Archive with the BioCASe Provider Software, it becomes automatically available for download on the BioCASe installation after the process has finished. The DwCA Creator can download the archive directly from this location. So if you're BioCASe installation is accessible from the machine you're running the prototype on, you can spare the trouble of moving the file manually and provide the URL instead of a local file name. To get the download URL for an ABCD Archive you've created, find the archive in question in the list of existing archives in the configuration tool (see [[Archiving#XML_Archives|here]] for more on that). Right-click on the Download link and choose ''Copy link''. Then open a terminal window for the creator tool and provide the link intead of a file name: <pre>./dwca.sh http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command: <pre>dwca.bat http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> ===Output=== The DwCA(s) created can be found in the folder ''output'' created in the base directory (''dwca''). Remember that there will be one DwCA per dataset. So depending upon how many datasets are stored in the source ABCD archive, there will be one or several files in the output folder. On Linux and MacOS, the shell script ''dwca.sh'' will zip all files that make up one DarwinCore Archive into an archive file named after the dataset title. So you could send this files directly to a potential DwCA consumer: [[File:dwcaOutputFolderLinux.png]] On Windows, the batch file ''dwca.bat'' of the prototype doesn't do that zipping. So if you go to the output file after the conversion has finished, you will see several text files and XML documents, all named after the pattern ''<dataset> <filename>''. So for the example above, the output folder would look like this: [[File:dwcaOutputFolderWindows.png]] In order to create a DwCA that can be sent to a consumer, please do the follwing for each ''<dataset>'': # Note down the dataset title, # Remove the dataset title in the file names (''<dataset> <filename>'' becomes ''<filename>''), # Create a zip named ''<dataset>.zip'' with all the files you just renamed. This lack of functionality is due to the limited capabilities of Windows batch files. Once integrated into the Provider Software, you won't need to zip up files manually. ===Trouble Shooting=== The default settings should be working for small and medium-sized collections (up to 500,000 records). If the processing is cancelled with an OutOfMemoryError, you should try the follwing: '''Increasing the maximum memory used by the transformer''' Per default, the transformer will use up to 1GB of RAM for the transformation process. In case of an OutOfMemoryError, you should increase this limit to 2GB. Don't be stingy with memory, since it will be only used during the tranformation process and be freed afterwards. Moreover, this is only the upper limit, so actual memory usage might well stay below this limit. To allow the transformer to use more memory, go to the folder ''kettle'' of your installation. Open the file ''kitchen.sh'' (Linux/MacOS) or ''kitchen.bat'' (Windows) in a text editor. Use the ''Find'' feature of the editor to go to the spot in file that specifies the memory limit. Replace the value ''1024'' with the amount of memory you want to allow the transformation to use, so e.g. ''2048'' for 2GB: [[File:dwcaSettingMemory.png]] Save the file and restart the transformation. '''Lowering the number of records''' 52840dfddb55a935ec10251bbad5dfd4cc62090c 437 436 2011-11-22T15:57:34Z JoergHoletschek 3 /* Trouble Shooting */ wikitext text/x-wiki With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Archives using the ABCD 2.06 schema (dubbed as “ABCD dumps”) can be converted into one or several DarwinCore Archives in a subsequent step. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than the ABCD dump it was created from. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:archArchivingPage.png|border]] ===Usage=== In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create a new Archive</small> allows you to create a new archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:archProcess.png|border]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== XML Archives that use the ABCD 2.06 schema can be converted into DarwinCore Archives. Currently, this feature is only available as a standalone command-line tool. Once thoroughly tested, it will be intregrated into the BioCASe Provider Software and can be trigged from the configuration tool. ===Requirements=== Since this tool is currently a standalone application, it doesn't need to be run on the same machine as the BioCASe Provider Software. But of course it can - if you're running BioCASe on a publicly available server and plan to use DwCAs for production, the BioCASe server is definitely the right place. * Java Runtime Environment 1.5 or later installed * 2+G of RAM available * 100M free disk space * One or several ABCD2.06 Archives (created with BPS3+) ===Installation=== # If you don't have a Java Runtime Environment (version 1.5 or later) installed on your machine, get one [http://www.java.com/de/download/ here]. # Download the DwCA Creator from the [http://www.biocase.org/products/provider_software/index.shtml#DwCA_Creator BioCASe Website] and unzip the archive file to your machine. This will create a folder ''dwca'' for the tool. # Test the installation by opening a terminal (''command prompt'' on Windows) and changing to the ''dwca'' folder. Start the tool by running the batch file ''dwca.bat'' on Windows or the shell script ''dwca.sh'' on Linux/MacOS. Since it doesn't do any processing, the test run should take only a few seconds and print some lines similar to following (you can ignore the warning ''No such file or directory'' in the last line): [[File:dwcaTest.png]] ===Usage=== The converter can be started with the scripts ''dwca.bat'' (on Windows) and ''dwca.sh'' (on Linux/MacOS) and usually takes one argument - the ABCD archive to be converted. The tool accepts both a file stored in the local file system and an archive available for download at a given URL. Let's assume you've created an ABCD2 dump for your collection. You've placed the archive file (named ''AlgaTerra_ABCD_2.06.zip'') in the same folder as the tool (''dwca''). To run the conversion on Linux, open a terminal window, change to the ''dwca'' folder and type in the following: <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command. Note that the base folder for the transformer is the ''kettle'' folder, so you need to specify ''..\'' to direct the transformer to its parent folder for finding the archive file (alternatively, you can put the archive file in the ''kettle'' subfolder and thus spare the ''..\''): <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip</pre> This will unzip the ABCD documents stored in the archive file, create the data files, EML document(s) and the archive descriptor(s): [[File:dwcaStartLocal.png]] Depending on the size of your collection (number of records), the amount of information published (number of concepts mapped) and the speed of your server (available memory, number of CPUs/cores), this will take from several seconds to some minutes. For large collections (millions of records), it can take up to hours. After each chunk of 50,000 records processed, a line indicating the progress will be printed, so you will be able to see if the tool is still running. However, independent of the amount of data to be processed, the last lines of the output will look similar to the following: [[File:dwcaEndLocal.png]] When creating an ABCD Archive with the BioCASe Provider Software, it becomes automatically available for download on the BioCASe installation after the process has finished. The DwCA Creator can download the archive directly from this location. So if you're BioCASe installation is accessible from the machine you're running the prototype on, you can spare the trouble of moving the file manually and provide the URL instead of a local file name. To get the download URL for an ABCD Archive you've created, find the archive in question in the list of existing archives in the configuration tool (see [[Archiving#XML_Archives|here]] for more on that). Right-click on the Download link and choose ''Copy link''. Then open a terminal window for the creator tool and provide the link intead of a file name: <pre>./dwca.sh http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command: <pre>dwca.bat http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> ===Output=== The DwCA(s) created can be found in the folder ''output'' created in the base directory (''dwca''). Remember that there will be one DwCA per dataset. So depending upon how many datasets are stored in the source ABCD archive, there will be one or several files in the output folder. On Linux and MacOS, the shell script ''dwca.sh'' will zip all files that make up one DarwinCore Archive into an archive file named after the dataset title. So you could send this files directly to a potential DwCA consumer: [[File:dwcaOutputFolderLinux.png]] On Windows, the batch file ''dwca.bat'' of the prototype doesn't do that zipping. So if you go to the output file after the conversion has finished, you will see several text files and XML documents, all named after the pattern ''<dataset> <filename>''. So for the example above, the output folder would look like this: [[File:dwcaOutputFolderWindows.png]] In order to create a DwCA that can be sent to a consumer, please do the follwing for each ''<dataset>'': # Note down the dataset title, # Remove the dataset title in the file names (''<dataset> <filename>'' becomes ''<filename>''), # Create a zip named ''<dataset>.zip'' with all the files you just renamed. This lack of functionality is due to the limited capabilities of Windows batch files. Once integrated into the Provider Software, you won't need to zip up files manually. ===Trouble Shooting=== The default settings should be working for small and medium-sized collections (up to 500,000 records). If the processing is cancelled with an OutOfMemoryError, you should try the follwing: '''Increasing the maximum memory used by the transformer''' Per default, the transformer will use up to 1GB of RAM for the transformation process. In case of an OutOfMemoryError, you should increase this limit to 2GB. Don't be stingy with memory, since it will be only used during the tranformation process and be freed afterwards. Moreover, this is only the upper limit, so actual memory usage might well stay below this limit. To allow the transformer to use more memory, go to the folder ''kettle'' of your installation. Open the file ''kitchen.sh'' (Linux/MacOS) or ''kitchen.bat'' (Windows) in a text editor. Use the ''Find'' feature of the editor to go to the spot in file that specifies the memory limit. Replace the value ''1024'' with the amount of memory you want to allow the transformation to use, so e.g. ''2048'' for 2GB: [[File:dwcaSettingMemory.png]] Save the file and restart the transformation. '''Lowering the number of records stored in memory'''<br> Per default, Kettle will store up to 100,000 records in memory before starting to use temporary files for sorting rows. You can set this threshold to a lower value to save some memory. However, be advised that this will result in heavy disk I/O on your machine, slowing down the transformation. Therefore you should first increase the memory limit before changing this setting. If you decide to do this, just add the desired maximim number of records to be stored in memory when invoking the transformer. So on Windows, type in On Windows, it would be like this: c94314e4af0d9d46792820058d05fdf0f5874d4d 438 437 2011-11-22T16:00:47Z JoergHoletschek 3 /* Trouble Shooting */ wikitext text/x-wiki With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Archives using the ABCD 2.06 schema (dubbed as “ABCD dumps”) can be converted into one or several DarwinCore Archives in a subsequent step. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than the ABCD dump it was created from. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:archArchivingPage.png|border]] ===Usage=== In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create a new Archive</small> allows you to create a new archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:archProcess.png|border]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== XML Archives that use the ABCD 2.06 schema can be converted into DarwinCore Archives. Currently, this feature is only available as a standalone command-line tool. Once thoroughly tested, it will be intregrated into the BioCASe Provider Software and can be trigged from the configuration tool. ===Requirements=== Since this tool is currently a standalone application, it doesn't need to be run on the same machine as the BioCASe Provider Software. But of course it can - if you're running BioCASe on a publicly available server and plan to use DwCAs for production, the BioCASe server is definitely the right place. * Java Runtime Environment 1.5 or later installed * 2+G of RAM available * 100M free disk space * One or several ABCD2.06 Archives (created with BPS3+) ===Installation=== # If you don't have a Java Runtime Environment (version 1.5 or later) installed on your machine, get one [http://www.java.com/de/download/ here]. # Download the DwCA Creator from the [http://www.biocase.org/products/provider_software/index.shtml#DwCA_Creator BioCASe Website] and unzip the archive file to your machine. This will create a folder ''dwca'' for the tool. # Test the installation by opening a terminal (''command prompt'' on Windows) and changing to the ''dwca'' folder. Start the tool by running the batch file ''dwca.bat'' on Windows or the shell script ''dwca.sh'' on Linux/MacOS. Since it doesn't do any processing, the test run should take only a few seconds and print some lines similar to following (you can ignore the warning ''No such file or directory'' in the last line): [[File:dwcaTest.png]] ===Usage=== The converter can be started with the scripts ''dwca.bat'' (on Windows) and ''dwca.sh'' (on Linux/MacOS) and usually takes one argument - the ABCD archive to be converted. The tool accepts both a file stored in the local file system and an archive available for download at a given URL. Let's assume you've created an ABCD2 dump for your collection. You've placed the archive file (named ''AlgaTerra_ABCD_2.06.zip'') in the same folder as the tool (''dwca''). To run the conversion on Linux, open a terminal window, change to the ''dwca'' folder and type in the following: <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command. Note that the base folder for the transformer is the ''kettle'' folder, so you need to specify ''..\'' to direct the transformer to its parent folder for finding the archive file (alternatively, you can put the archive file in the ''kettle'' subfolder and thus spare the ''..\''): <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip</pre> This will unzip the ABCD documents stored in the archive file, create the data files, EML document(s) and the archive descriptor(s): [[File:dwcaStartLocal.png]] Depending on the size of your collection (number of records), the amount of information published (number of concepts mapped) and the speed of your server (available memory, number of CPUs/cores), this will take from several seconds to some minutes. For large collections (millions of records), it can take up to hours. After each chunk of 50,000 records processed, a line indicating the progress will be printed, so you will be able to see if the tool is still running. However, independent of the amount of data to be processed, the last lines of the output will look similar to the following: [[File:dwcaEndLocal.png]] When creating an ABCD Archive with the BioCASe Provider Software, it becomes automatically available for download on the BioCASe installation after the process has finished. The DwCA Creator can download the archive directly from this location. So if you're BioCASe installation is accessible from the machine you're running the prototype on, you can spare the trouble of moving the file manually and provide the URL instead of a local file name. To get the download URL for an ABCD Archive you've created, find the archive in question in the list of existing archives in the configuration tool (see [[Archiving#XML_Archives|here]] for more on that). Right-click on the Download link and choose ''Copy link''. Then open a terminal window for the creator tool and provide the link intead of a file name: <pre>./dwca.sh http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command: <pre>dwca.bat http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> ===Output=== The DwCA(s) created can be found in the folder ''output'' created in the base directory (''dwca''). Remember that there will be one DwCA per dataset. So depending upon how many datasets are stored in the source ABCD archive, there will be one or several files in the output folder. On Linux and MacOS, the shell script ''dwca.sh'' will zip all files that make up one DarwinCore Archive into an archive file named after the dataset title. So you could send this files directly to a potential DwCA consumer: [[File:dwcaOutputFolderLinux.png]] On Windows, the batch file ''dwca.bat'' of the prototype doesn't do that zipping. So if you go to the output file after the conversion has finished, you will see several text files and XML documents, all named after the pattern ''<dataset> <filename>''. So for the example above, the output folder would look like this: [[File:dwcaOutputFolderWindows.png]] In order to create a DwCA that can be sent to a consumer, please do the follwing for each ''<dataset>'': # Note down the dataset title, # Remove the dataset title in the file names (''<dataset> <filename>'' becomes ''<filename>''), # Create a zip named ''<dataset>.zip'' with all the files you just renamed. This lack of functionality is due to the limited capabilities of Windows batch files. Once integrated into the Provider Software, you won't need to zip up files manually. ===Trouble Shooting=== The default settings should be working for small and medium-sized collections (up to 500,000 records). If the processing is cancelled with an OutOfMemoryError, you should try the follwing: '''Increasing the maximum memory used by the transformer''' Per default, the transformer will use up to 1GB of RAM for the transformation process. In case of an OutOfMemoryError, you should increase this limit to 2GB. Don't be stingy with memory, since it will be only used during the tranformation process and be freed afterwards. Moreover, this is only the upper limit, so actual memory usage might well stay below this limit. To allow the transformer to use more memory, go to the folder ''kettle'' of your installation. Open the file ''kitchen.sh'' (Linux/MacOS) or ''kitchen.bat'' (Windows) in a text editor. Use the ''Find'' feature of the editor to go to the spot in file that specifies the memory limit. Replace the value ''1024'' with the amount of memory you want to allow the transformation to use, so e.g. ''2048'' for 2GB: [[File:dwcaSettingMemory.png]] Save the file and restart the transformation. '''Lowering the number of records stored in memory'''<br> Per default, Kettle will store up to 100,000 records in memory before starting to use temporary files for sorting rows. You can set this threshold to a lower value to save some memory. However, be advised that this will result in heavy disk I/O on your machine, slowing down the transformation. Therefore you should first increase the memory limit before changing this setting. If you decide to do this, just add the desired maximim number of records to be stored in memory as a second parameter when invoking the transformer. So on Linux, type in <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip 10000</pre> in order to set the threshold to 10,000 records. On a Windows machine, type in following: <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip 10000</pre> 25d10df594f9ef0707e4c48bf0a2aa2597d32ec8 439 438 2011-11-22T16:03:51Z JoergHoletschek 3 /* Trouble Shooting */ wikitext text/x-wiki With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Archives using the ABCD 2.06 schema (dubbed as “ABCD dumps”) can be converted into one or several DarwinCore Archives in a subsequent step. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than the ABCD dump it was created from. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:archArchivingPage.png|border]] ===Usage=== In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create a new Archive</small> allows you to create a new archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:archProcess.png|border]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== XML Archives that use the ABCD 2.06 schema can be converted into DarwinCore Archives. Currently, this feature is only available as a standalone command-line tool. Once thoroughly tested, it will be intregrated into the BioCASe Provider Software and can be trigged from the configuration tool. ===Requirements=== Since this tool is currently a standalone application, it doesn't need to be run on the same machine as the BioCASe Provider Software. But of course it can - if you're running BioCASe on a publicly available server and plan to use DwCAs for production, the BioCASe server is definitely the right place. * Java Runtime Environment 1.5 or later installed * 2+G of RAM available * 100M free disk space * One or several ABCD2.06 Archives (created with BPS3+) ===Installation=== # If you don't have a Java Runtime Environment (version 1.5 or later) installed on your machine, get one [http://www.java.com/de/download/ here]. # Download the DwCA Creator from the [http://www.biocase.org/products/provider_software/index.shtml#DwCA_Creator BioCASe Website] and unzip the archive file to your machine. This will create a folder ''dwca'' for the tool. # Test the installation by opening a terminal (''command prompt'' on Windows) and changing to the ''dwca'' folder. Start the tool by running the batch file ''dwca.bat'' on Windows or the shell script ''dwca.sh'' on Linux/MacOS. Since it doesn't do any processing, the test run should take only a few seconds and print some lines similar to following (you can ignore the warning ''No such file or directory'' in the last line): [[File:dwcaTest.png]] ===Usage=== The converter can be started with the scripts ''dwca.bat'' (on Windows) and ''dwca.sh'' (on Linux/MacOS) and usually takes one argument - the ABCD archive to be converted. The tool accepts both a file stored in the local file system and an archive available for download at a given URL. Let's assume you've created an ABCD2 dump for your collection. You've placed the archive file (named ''AlgaTerra_ABCD_2.06.zip'') in the same folder as the tool (''dwca''). To run the conversion on Linux, open a terminal window, change to the ''dwca'' folder and type in the following: <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command. Note that the base folder for the transformer is the ''kettle'' folder, so you need to specify ''..\'' to direct the transformer to its parent folder for finding the archive file (alternatively, you can put the archive file in the ''kettle'' subfolder and thus spare the ''..\''): <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip</pre> This will unzip the ABCD documents stored in the archive file, create the data files, EML document(s) and the archive descriptor(s): [[File:dwcaStartLocal.png]] Depending on the size of your collection (number of records), the amount of information published (number of concepts mapped) and the speed of your server (available memory, number of CPUs/cores), this will take from several seconds to some minutes. For large collections (millions of records), it can take up to hours. After each chunk of 50,000 records processed, a line indicating the progress will be printed, so you will be able to see if the tool is still running. However, independent of the amount of data to be processed, the last lines of the output will look similar to the following: [[File:dwcaEndLocal.png]] When creating an ABCD Archive with the BioCASe Provider Software, it becomes automatically available for download on the BioCASe installation after the process has finished. The DwCA Creator can download the archive directly from this location. So if you're BioCASe installation is accessible from the machine you're running the prototype on, you can spare the trouble of moving the file manually and provide the URL instead of a local file name. To get the download URL for an ABCD Archive you've created, find the archive in question in the list of existing archives in the configuration tool (see [[Archiving#XML_Archives|here]] for more on that). Right-click on the Download link and choose ''Copy link''. Then open a terminal window for the creator tool and provide the link intead of a file name: <pre>./dwca.sh http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command: <pre>dwca.bat http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> ===Output=== The DwCA(s) created can be found in the folder ''output'' created in the base directory (''dwca''). Remember that there will be one DwCA per dataset. So depending upon how many datasets are stored in the source ABCD archive, there will be one or several files in the output folder. On Linux and MacOS, the shell script ''dwca.sh'' will zip all files that make up one DarwinCore Archive into an archive file named after the dataset title. So you could send this files directly to a potential DwCA consumer: [[File:dwcaOutputFolderLinux.png]] On Windows, the batch file ''dwca.bat'' of the prototype doesn't do that zipping. So if you go to the output file after the conversion has finished, you will see several text files and XML documents, all named after the pattern ''<dataset> <filename>''. So for the example above, the output folder would look like this: [[File:dwcaOutputFolderWindows.png]] In order to create a DwCA that can be sent to a consumer, please do the follwing for each ''<dataset>'': # Note down the dataset title, # Remove the dataset title in the file names (''<dataset> <filename>'' becomes ''<filename>''), # Create a zip named ''<dataset>.zip'' with all the files you just renamed. This lack of functionality is due to the limited capabilities of Windows batch files. Once integrated into the Provider Software, you won't need to zip up files manually. ===Trouble Shooting=== If you run into problems when using this tool, please [[Special:Contact|contact us]] for help. The default settings of the transormation should be working for small and medium-sized collections (up to 500,000 records). If the processing is cancelled with an OutOfMemoryError, you should try the follwing: '''Increasing the maximum memory used by the transformer'''<br> Per default, the transformer will use up to 1GB of RAM for the transformation process. In case of an OutOfMemoryError, you should increase this limit to 2GB. Don't be stingy with memory, since it will be only used during the tranformation process and be freed afterwards. Moreover, this is only the upper limit, so actual memory usage might well stay below this limit. To allow the transformer to use more memory, go to the folder ''kettle'' of your installation. Open the file ''kitchen.sh'' (Linux/MacOS) or ''kitchen.bat'' (Windows) in a text editor. Use the ''Find'' feature of the editor to go to the spot in file that specifies the memory limit. Replace the value ''1024'' with the amount of memory you want to allow the transformation to use, so e.g. ''2048'' for 2GB: [[File:dwcaSettingMemory.png]] Save the file and restart the transformation. '''Lowering the number of records stored in memory'''<br> Per default, Kettle will store up to 100,000 records in memory before starting to use temporary files for sorting rows. You can set this threshold to a lower value to save some memory. However, be advised that this will result in heavy disk I/O on your machine, slowing down the transformation. Therefore you should first increase the memory limit before changing this setting. If you decide to do this, just add the desired maximim number of records to be stored in memory as a second parameter when invoking the transformer. So on Linux, type in <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip 10000</pre> in order to set the threshold to 10,000 records. On a Windows machine, type in following: <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip 10000</pre> a3014672591aa3e1cddd23d2f379b744a2243b80 6 85 418 2011-11-22T12:59:53Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 86 422 2011-11-22T13:43:08Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 87 423 2011-11-22T13:43:30Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 88 428 2011-11-22T14:24:27Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 89 429 2011-11-22T14:24:50Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 90 435 2011-11-22T15:49:54Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 0 23 440 392 2011-12-08T14:01:42Z JoergHoletschek 3 /* Version 3.0RC [2011-11-01] */ wikitext text/x-wiki == Current Version 3 == ===Version 3.0RC [2011-11-01]=== * New Archiving feature * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 67e065295312f06901c4cf6bca2b09b52888abac 441 440 2011-12-08T14:03:56Z JoergHoletschek 3 /* Version 3.0RC [2011-11-01] */ wikitext text/x-wiki == Current Version 3 == ===Version 3.0RC2 [2011-11-24]=== * New Archiving feature * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 79142b7e045efca9d0928102a0fb0cdd483ca3d8 442 441 2011-12-08T15:02:02Z JoergHoletschek 3 /* Version 3.0RC2 [2011-11-24] */ wikitext text/x-wiki == Current Version 3 == ===Version 3.0RC2 [2011-11-24]=== * New feature: XML Archives (integrated into UI), DwC Archives as a separate application * List of table/column names now also available for Oracle databases * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 597776546ef42d28f894eb28556dd26148160d71 444 442 2011-12-08T15:20:30Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == ===Version 3.0RC2 [2011-11-24]=== * New feature: XML Archives (integrated into UI), DwC Archives as a separate application * List of table/column names now also available for Oracle databases * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 0fca17f77ca6e985020b017a1ddb9038d9d6278e 447 444 2011-12-14T12:05:18Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives as a separate application * List of table/column names now also available for Oracle databases * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool b0d919a163ef90b66a4108b9b421fc902fe23c68 448 447 2011-12-14T12:06:13Z JoergHoletschek 3 /* Version 3.0 [2011-12-14] */ wikitext text/x-wiki == Current Version 3 == ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives as a separate application * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 4a662c19eb49313404cf6e7d6b467a51cfdad721 449 448 2011-12-14T12:07:09Z JoergHoletschek 3 /* Version 3.0 [2011-12-14] */ wikitext text/x-wiki == Current Version 3 == ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool c5ac9feb2e5e129a7446f2862e5d04d074954ba0 450 449 2012-01-03T16:43:23Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == ===Upcoming=== * Bugfix in library test page (for the case that Apache has different access privileges than Python) ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 1630c435794bcff0cad766708e87431a2afbb934 451 450 2012-01-04T12:16:39Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == ===Upcoming=== * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * Bugfix in library test page (for the case that Apache has different access privileges than Python) ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 19f536a8f9d1a481f6695ab3f5c22a70130a6ec1 452 451 2012-01-04T15:33:57Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == ===Upcoming=== * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources * Bugfix in library test page (for the case that Apache has different access privileges than Python) ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 5a4a4fd696aee80627873231d5ed64a88ee3b45f 458 452 2012-01-05T17:49:52Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == ===Upcoming=== * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 616ae732e14a46dfd9ba647af97fa1b19f3455e9 0 52 443 393 2011-12-08T15:18:37Z JoergHoletschek 3 /* Installation */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I cannot find the installation package of the database connectivity module for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later of psycopg2. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation to the ''config'' folder of your new installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. You can then edit new installation and test the web services. ===I've updated my installation from a version prior to 2.6. Now, when I enter the DB structure, some of the table and column names are gone, the same happens in the mapping editor. I use MySQL, SQL Server, Postgres, Access or Oracle.=== Starting with version 2.6, BioCASe retrieves table and column names from the database for the mentioned database management systems. When you did the setup with an earlier version of BioCASe, you entered table and column names manually - and probably didn't care for upper/lower cases. Since some DB management systems are case-sensitive, BioCASe can't ignore the differences and won't find these technically incorrect names in the lists retrieved from the database. The respective drop down boxes will therefore be empty. If that happens, just select the correct names from the drop down lists. ===When I try to create an archive, I get a "Permission denied" error. Why is that?=== For creating archives, you need to grant write permissions for the folders ''archive'' and ''www/downloads'' of your BioCASe installation to Python. You can check if permissions are set correctly in the test libs page (Start --> Utilities --> Test Libs). == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other only by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. == Trouble Shooting == ===I’ve forgotten the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. 6976542f58797d8029e5fad833f840d575199420 446 443 2011-12-08T15:34:38Z JoergHoletschek 3 /* I've updated my installation from a version prior to 2.6. Now, when I enter the DB structure, some of the table and column names are gone, the same happens in the mapping editor. I use MySQL, SQL Server, Postgres, Access or Oracle. */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I cannot find the installation package of the database connectivity module for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later of psycopg2. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation to the ''config'' folder of your new installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. You can then edit new installation and test the web services. ===I've updated my installation from a version prior to 2.6. Now, when I enter the DB structure, some of the table and column names are gone, the same happens in the mapping editor.=== Starting with version 2.6, BioCASe retrieves table and column names from the database for some database management systems (namely MySQL, SQL Server, Postgres, Foxpro, Access and Oracle). In older versions, you had to enter table/column names manually, and when you did the setup with such an early version, you probably didn't care for upper/lower cases. Since some DB management systems are case-sensitive, BioCASe can't ignore the differences and won't find these technically incorrect names in the lists retrieved from the database. The respective drop down boxes will therefore be empty. If that happens, just select the correct names from the drop down lists. ===When I try to create an archive, I get a "Permission denied" error. Why is that?=== For creating archives, you need to grant write permissions for the folders ''archive'' and ''www/downloads'' of your BioCASe installation to Python. You can check if permissions are set correctly in the test libs page (Start --> Utilities --> Test Libs). == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other only by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. == Trouble Shooting == ===I’ve forgotten the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. 958bdb7152566d816ce624b4e8146eccc5e32368 453 446 2012-01-05T13:08:26Z JoergHoletschek 3 /* Debugging */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I cannot find the installation package of the database connectivity module for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later of psycopg2. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation to the ''config'' folder of your new installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. You can then edit new installation and test the web services. ===I've updated my installation from a version prior to 2.6. Now, when I enter the DB structure, some of the table and column names are gone, the same happens in the mapping editor.=== Starting with version 2.6, BioCASe retrieves table and column names from the database for some database management systems (namely MySQL, SQL Server, Postgres, Foxpro, Access and Oracle). In older versions, you had to enter table/column names manually, and when you did the setup with such an early version, you probably didn't care for upper/lower cases. Since some DB management systems are case-sensitive, BioCASe can't ignore the differences and won't find these technically incorrect names in the lists retrieved from the database. The respective drop down boxes will therefore be empty. If that happens, just select the correct names from the drop down lists. ===When I try to create an archive, I get a "Permission denied" error. Why is that?=== For creating archives, you need to grant write permissions for the folders ''archive'' and ''www/downloads'' of your BioCASe installation to Python. You can check if permissions are set correctly in the test libs page (Start --> Utilities --> Test Libs). == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other only by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===When testing the web service, I get a negative number for "RecordsDropped" in the response document. What does that mean?=== ===I'm using pymssql to connect to a SQL Server database. Char(n) and Varchar(n) strings get truncated to 255 characters - what's wrong?=== This is known limitation of TDS protocol used by pymssql. Just change the data type for columns storing long string to TEXT or create a view that converts Char/Varchar columns to TEXT columns on the fly to work around this issue. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. == Trouble Shooting == ===I’ve forgotten the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. 8451d8aa87eb998119ac8f8aeaba3a3cdf1df67d 454 453 2012-01-05T13:16:04Z JoergHoletschek 3 /* Debugging */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I cannot find the installation package of the database connectivity module for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later of psycopg2. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation to the ''config'' folder of your new installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. You can then edit new installation and test the web services. ===I've updated my installation from a version prior to 2.6. Now, when I enter the DB structure, some of the table and column names are gone, the same happens in the mapping editor.=== Starting with version 2.6, BioCASe retrieves table and column names from the database for some database management systems (namely MySQL, SQL Server, Postgres, Foxpro, Access and Oracle). In older versions, you had to enter table/column names manually, and when you did the setup with such an early version, you probably didn't care for upper/lower cases. Since some DB management systems are case-sensitive, BioCASe can't ignore the differences and won't find these technically incorrect names in the lists retrieved from the database. The respective drop down boxes will therefore be empty. If that happens, just select the correct names from the drop down lists. ===When I try to create an archive, I get a "Permission denied" error. Why is that?=== For creating archives, you need to grant write permissions for the folders ''archive'' and ''www/downloads'' of your BioCASe installation to Python. You can check if permissions are set correctly in the test libs page (Start --> Utilities --> Test Libs). == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other only by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===When testing the web service, I get a negative number for "RecordsDropped" in the response document. What does that mean?=== It means you have an error in your mapping. Most probably you've mapped a non-repeatable ABCD concept to a table/column in your database that holds several entries per unit (occurrence) record. The BioCASe Provider Software will duplicate the respective unit records to be able to create valid ABCD documents - which is indicated by a negative number of "records dropped". Make sure the cardinality of your database relations match the structure (in respects of element repeatability) of ABCD to avoid this problem. ===I'm using pymssql to connect to a SQL Server database. Char(n) and Varchar(n) strings get truncated to 255 characters - what's wrong?=== This is known limitation of TDS protocol used by pymssql. Just change the data type for columns storing long string to TEXT or create a view that converts Char/Varchar columns to TEXT columns on the fly to work around this issue. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. == Trouble Shooting == ===I’ve forgotten the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. 7e0a5e972e6eba66101a04d1cb7f6d0d668bcc98 455 454 2012-01-05T13:17:09Z JoergHoletschek 3 /* When testing the web service, I get a negative number for "RecordsDropped" in the response document. What does that mean? */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I cannot find the installation package of the database connectivity module for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later of psycopg2. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation to the ''config'' folder of your new installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. You can then edit new installation and test the web services. ===I've updated my installation from a version prior to 2.6. Now, when I enter the DB structure, some of the table and column names are gone, the same happens in the mapping editor.=== Starting with version 2.6, BioCASe retrieves table and column names from the database for some database management systems (namely MySQL, SQL Server, Postgres, Foxpro, Access and Oracle). In older versions, you had to enter table/column names manually, and when you did the setup with such an early version, you probably didn't care for upper/lower cases. Since some DB management systems are case-sensitive, BioCASe can't ignore the differences and won't find these technically incorrect names in the lists retrieved from the database. The respective drop down boxes will therefore be empty. If that happens, just select the correct names from the drop down lists. ===When I try to create an archive, I get a "Permission denied" error. Why is that?=== For creating archives, you need to grant write permissions for the folders ''archive'' and ''www/downloads'' of your BioCASe installation to Python. You can check if permissions are set correctly in the test libs page (Start --> Utilities --> Test Libs). == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other only by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===When testing the web service, I get a negative number for "RecordsDropped" in the response document. What does that mean?=== Same problem as above - it means you have an error in your mapping. Most probably you've mapped a non-repeatable ABCD concept to a table/column in your database that holds several entries per unit (occurrence) record. The BioCASe Provider Software will duplicate the respective unit records to be able to create valid ABCD documents - which is indicated by a negative number of "records dropped". Make sure the cardinality of your database relations match the structure (in respects of element repeatability) of ABCD to avoid this problem. ===I'm using pymssql to connect to a SQL Server database. Char(n) and Varchar(n) strings get truncated to 255 characters - what's wrong?=== This is known limitation of TDS protocol used by pymssql. Just change the data type for columns storing long string to TEXT or create a view that converts Char/Varchar columns to TEXT columns on the fly to work around this issue. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. == Trouble Shooting == ===I’ve forgotten the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. 1951931fdf5fd1dff6ddbaa436a0dfcce90e5b3e 456 455 2012-01-05T13:17:32Z JoergHoletschek 3 /* When testing the web service, I get a negative number for "RecordsDropped" in the response document. What does that mean? */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I cannot find the installation package of the database connectivity module for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later of psycopg2. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation to the ''config'' folder of your new installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. You can then edit new installation and test the web services. ===I've updated my installation from a version prior to 2.6. Now, when I enter the DB structure, some of the table and column names are gone, the same happens in the mapping editor.=== Starting with version 2.6, BioCASe retrieves table and column names from the database for some database management systems (namely MySQL, SQL Server, Postgres, Foxpro, Access and Oracle). In older versions, you had to enter table/column names manually, and when you did the setup with such an early version, you probably didn't care for upper/lower cases. Since some DB management systems are case-sensitive, BioCASe can't ignore the differences and won't find these technically incorrect names in the lists retrieved from the database. The respective drop down boxes will therefore be empty. If that happens, just select the correct names from the drop down lists. ===When I try to create an archive, I get a "Permission denied" error. Why is that?=== For creating archives, you need to grant write permissions for the folders ''archive'' and ''www/downloads'' of your BioCASe installation to Python. You can check if permissions are set correctly in the test libs page (Start --> Utilities --> Test Libs). == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other only by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===I've mapped ABCD. When testing the web service, I get a negative number for "RecordsDropped" in the response document. What does that mean?=== Same problem as above - it means you have an error in your mapping. Most probably you've mapped a non-repeatable ABCD concept to a table/column in your database that holds several entries per unit (occurrence) record. The BioCASe Provider Software will duplicate the respective unit records to be able to create valid ABCD documents - which is indicated by a negative number of "records dropped". Make sure the cardinality of your database relations match the structure (in respects of element repeatability) of ABCD to avoid this problem. ===I'm using pymssql to connect to a SQL Server database. Char(n) and Varchar(n) strings get truncated to 255 characters - what's wrong?=== This is known limitation of TDS protocol used by pymssql. Just change the data type for columns storing long string to TEXT or create a view that converts Char/Varchar columns to TEXT columns on the fly to work around this issue. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. == Trouble Shooting == ===I’ve forgotten the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. 5b5a50a578dc25a0ae799acd5596f9d0b5621d8e 457 456 2012-01-05T13:17:43Z JoergHoletschek 3 /* I've mapped ABCD. When testing the web service, I get a negative number for "RecordsDropped" in the response document. What does that mean? */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I cannot find the installation package of the database connectivity module for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later of psycopg2. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation to the ''config'' folder of your new installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. You can then edit new installation and test the web services. ===I've updated my installation from a version prior to 2.6. Now, when I enter the DB structure, some of the table and column names are gone, the same happens in the mapping editor.=== Starting with version 2.6, BioCASe retrieves table and column names from the database for some database management systems (namely MySQL, SQL Server, Postgres, Foxpro, Access and Oracle). In older versions, you had to enter table/column names manually, and when you did the setup with such an early version, you probably didn't care for upper/lower cases. Since some DB management systems are case-sensitive, BioCASe can't ignore the differences and won't find these technically incorrect names in the lists retrieved from the database. The respective drop down boxes will therefore be empty. If that happens, just select the correct names from the drop down lists. ===When I try to create an archive, I get a "Permission denied" error. Why is that?=== For creating archives, you need to grant write permissions for the folders ''archive'' and ''www/downloads'' of your BioCASe installation to Python. You can check if permissions are set correctly in the test libs page (Start --> Utilities --> Test Libs). == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other only by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===I've mapped ABCD2. When testing the web service, I get a negative number for "RecordsDropped" in the response document. What does that mean?=== Same problem as above - it means you have an error in your mapping. Most probably you've mapped a non-repeatable ABCD concept to a table/column in your database that holds several entries per unit (occurrence) record. The BioCASe Provider Software will duplicate the respective unit records to be able to create valid ABCD documents - which is indicated by a negative number of "records dropped". Make sure the cardinality of your database relations match the structure (in respects of element repeatability) of ABCD to avoid this problem. ===I'm using pymssql to connect to a SQL Server database. Char(n) and Varchar(n) strings get truncated to 255 characters - what's wrong?=== This is known limitation of TDS protocol used by pymssql. Just change the data type for columns storing long string to TEXT or create a view that converts Char/Varchar columns to TEXT columns on the fly to work around this issue. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. == Trouble Shooting == ===I’ve forgotten the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. d308851aa042257c1cc44c75ee5fbacaa3b049e5 459 457 2012-01-10T10:07:32Z JoergHoletschek 3 /* Debugging */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I cannot find the installation package of the database connectivity module for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later of psycopg2. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation to the ''config'' folder of your new installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. You can then edit new installation and test the web services. ===I've updated my installation from a version prior to 2.6. Now, when I enter the DB structure, some of the table and column names are gone, the same happens in the mapping editor.=== Starting with version 2.6, BioCASe retrieves table and column names from the database for some database management systems (namely MySQL, SQL Server, Postgres, Foxpro, Access and Oracle). In older versions, you had to enter table/column names manually, and when you did the setup with such an early version, you probably didn't care for upper/lower cases. Since some DB management systems are case-sensitive, BioCASe can't ignore the differences and won't find these technically incorrect names in the lists retrieved from the database. The respective drop down boxes will therefore be empty. If that happens, just select the correct names from the drop down lists. ===When I try to create an archive, I get a "Permission denied" error. Why is that?=== For creating archives, you need to grant write permissions for the folders ''archive'' and ''www/downloads'' of your BioCASe installation to Python. You can check if permissions are set correctly in the test libs page (Start --> Utilities --> Test Libs). == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other only by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===I've mapped ABCD2. When testing the web service, I get a negative number for "RecordsDropped" in the response document. What does that mean?=== Same problem as above - it means you have an error in your mapping. Most probably you've mapped a non-repeatable ABCD concept to a table/column in your database that holds several entries per unit (occurrence) record. The BioCASe Provider Software will duplicate the respective unit records to be able to create valid ABCD documents - which is indicated by a negative number of "records dropped". Make sure the cardinality of your database relations match the structure (in respects of element repeatability) of ABCD to avoid this problem. ===I'm using pymssql to connect to a SQL Server database. Char(n) and Varchar(n) strings get truncated to 255 characters - what's wrong?=== This is known limitation of TDS protocol used by pymssql. Just change the data type for columns storing long string to TEXT or create a view that converts Char/Varchar columns to TEXT columns on the fly to work around this issue. ===I'm using pymssql to connect to a SQL Server database, BioCASe is running on a Linux distribution. When I do a Search or Scan request, I don't get any results (0 hits), even though there are definitely records in the database.=== That problem occurs on old versions of pymssql (1.0.2 or earlier) on Linux distributions. Update to version 2.0 or later to solve this. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. == Trouble Shooting == ===I’ve forgotten the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. ace999355e9eda4fbea71ad7e639b8d725a5d96b 0 24 445 369 2011-12-08T15:29:53Z JoergHoletschek 3 /* Saving, Editing an Existing Setup */ wikitext text/x-wiki This tutorial explains in detail how to set up a datasource. == A BioCASe Data Source == Publishing a database with BioCASe means it will be made accessible through a BioCASe web service, which will be used later by the biodiversity network(s) you want to join to access to your data. For example, the GBIF Harvester will send a BioCASe Scan request to that web service to get a list of all scientific names kept in the database, and subsequently several BioCASe Search requests to load all records for these names. For each database you want to publish, you need to set up a BioCASe data source (do not mix that up with ODBC data sources on Windows machines). The resulting BioCASe web service is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source. So if you made your installation available at http://www.foobar.org/biocase during the installation process and set up a data source named Herbar, the URL of the BioCASe web service would be http://www.foobar.org/biocase/pywrapper.cgi?dsa=Herbar (data source names can be case sensitive, depending on your server’s operating system). == Creating a New Data Source == The start page of the BPS will always list the data sources currently configured, that means the web services offered by that BioCASe installation. On a fresh installation, it will be empty: [[File:instBiocaseStartpage.png|500px]] Creating a new data source can be done in the configuration tool. Choose <small>Config Tool</small> on the start page, then <small>System Administration</small>. Log in with the password you set during the installation process (if you didn’t change that yet, the default is ''ACDC''). In the configuration screen, the sections <small>Data Sources</small> and <small>Recycle Bin</small> can be used to create, remove und restore a data source: [[File:dsaCreateDatasourceEmpty.png|500px|border]] For now, we want to create a new, empty data source. Enter a name – without any spaces – in the text box and click <small>Create DSA</small> (remember that the datasource name is case sensitive on Linux machines). After a second, it will appear in the list of existing data sources with a clickable link. By doing this, you created the first BioCASe web service for your installation. Clicking <small>Remove</small> will uninstall this service again (putting it in the recycle bin, so you can restore it later). Tip: If you select <small>abcdmetadata</small> in the template drop-down list, your data source will be created with an ABCD mapping for all the metadata elements stored in the metadata table mentioned in the [[Preparation]] guide. So if you did use the templates in ''sampledb/abcdmetadata'' to create the metadata table in your database, you can use this template to spare the hassle of mapping this table and start right away with your occurrence data. == Setting up the database connection == Clicking on the data source you’ve just created will take you to the datasource configuration tool. Note that the <small>Schemas</small> section will be empty if you’ve created an empty data source or list two schemas (ABCD 1.2 and 2.06) if you created the datasource based on the <small>abcdmetadata</small> template: [[File:dsaOverview.png|450px]] The link in the <small>Database Connection</small> section will take you to the connection parameters dialog. Fill in the values for your database, press <small>Save</small> and the connection status should turn into a nice green <small>OK</small>. '''DBMS:''' Select the entry appropriate for your collection. The BPS can directly connect to database servers (MySQL, Postgres, DB2, SQL Server, and Oracle) and to Desktop-DBMS like Access and FoxPro through ODBC. ODBC is only available on Windows, so if you plan to use it, BioCASe must be installed on a Windows Server. '''Host:''' Enter the IP address of the database server. If you’re connecting through ODBC, you must leave this empty. Instead, use Windows’ ODBC Administrator to set up a System (!) datasource; the server or file name of the database will be stored in the ODBC configuration. '''Database:''' Enter the name of the ODBC datasource if you’re using ODBC, or the name of the database if you’re directly connecting to a server. '''User/Password:''' Database credentials to be used. As written in the [[Preparation]] guide, you should create a user with just read access for this. '''Encoding:''' The encoding of your database. If you’re not sure, leave the default, you can change it if you run into problems later. If you know your database is using a different encoding than latin-1 (which is typically the default for western languages), maybe UTF-8, select the appropriate entry in the drop-down list. [[File:dsaDBConnection.png|450px]] Once you’ve entered the correct values, press <small>Save</small>. Click <small>Overview</small> in the link list at the top of the page to return to the datasource configuration page. If the connection status doesn’t turn into <small>OK</small> (and doesn’t after you’ve returned to the overview page), please read the [[Debugging]] guide about how to trace the problem. == Setting up the Database Structure == Clicking on <small>Edit DB Structure</small> on the datasource overview page will show the DB structure setup. Make sure the connection status at the top of the page still is <small>OK</small>. In a first step, you will add all tables/views that hold information you want to publish with BioCASe. You can ignore all database tables storing only data you don’t plan to map to ABCD. Same holds true if you use cache tables (as explained in the [[Preparation]] guide): Only the cache tables need to be set up on the database structure page. In a second step, you will specify the foreign key references between these tables. === Adding Tables/Views with their Primary Keys === To add a table or view, just select it in the drop-down list. The name of the table will be filled in the <small>Alias</small> text box (you will use this alias later in the mapping process), feel free to change it into something that describes the contents of that tables (for example ''specimen'' for the root specimen table). Press <small>Add</small> to insert the table into the list. Before continuing to the next table, select the primary key in the list of attributes under <small>Primary Key Attribute(s)</small>. In case the data type is a string and not an integer, select <small>text</small> from the drop-down list. Press <small>Add</small> to add the primary key. If the primary key consists of several columns, you can add several primary key attributes. Redo these steps (adding the tables first, then the primary key/keys) for all tables you want to publish data from. Always remember to set the correct primary key column(s), even if they’re not defined as primary keys in the database.<br> '''<u>Important Note:</u>''' Tables without a primary key cannot be used by the Provider Software and will be removed upon saving!''' === Specifying Foreign Key References === Once you’ve added all tables, you can define the relations between these tables by defining foreign keys. In order to do this, go to the <small>Foreign Key(s)</small> column of the table that holds a foreign key reference to another table, select the alias of the referenced table and press <small>Add</small>. Select the attribute that stores the foreign key and change the data type if it is not integer, then press <small>Add</small>. You must specify as many attributes for the foreign keys as the primary key of the referenced table has, otherwise the Provider Software won’t be able to create a valid join between the tables. Also, they must be in the same order and have identical data types. Redo this for all tables until you’ve specified all foreign key references. If you’ve installed the Graphviz package, the Provider Software will create a graph visualizing the current database structure. Each table will be visible as a node with the primary key(s) listed. References between tables will be represented by arrows, the foreign key(s) listed in the referring table: [[File:dsaDatabaseStructure_AlgenEngels.png|800px]] === Saving, Editing an Existing Setup === You can change aliases of tables you’ve already added to the list. To do so, go to the <small>Alias</small> text box of that table and type in a different name. Upon exiting the text box, all references to this alias will be updated in the table setup; also the graph will reflect the changes. But caution: Schema mappings use these aliases to bind schema elements to tables and columns, if you change an alias you’ve already used in a schema mapping, you will break this binding! So it is advisable to decide for meaningful alias names and keep them stable once you’ve started the mapping process. Once you’ve added a table/view to the table list, you can’t change the underlying table, since this would invalidate all the primary and foreign key attributes. If you’ve added the wrong table, remove it by using the trash can symbol next to <small>Delete Alias</small> and add the correct table/view. If you’ve finished the table setup and the graph displays the model of your database correctly, save the configuration by pressing <small>Save</small> at the bottom of the page. This is important; navigating away from the page without saving will cause your changes to be lost (even though you will be asked for confirmation)! It is advisable to save the table setup every now and then, for example once you’ve added all tables and primary keys. If you edit an existing table setup and happen to realize you’ve messed it up, you can revert to the state when you’ve pressed <small>Save</small> for the last time by using the <small>Revert</small> button at the bottom. But be careful: All unsaved changes will be lost. '''<u>Important Note:</u>''' The retrieval of tables/view/column names is currently only supported for the following DBMS: Microsoft SQL Server, MySQL, Oracle, Access, Postgres and FoxPro. If you use a different DBMS, you will see a simple text box instead of a drop-down-list, where you’ll have to type in the name of the table or column you want to add. 53666e6f81c8a567625f15acc9e5ac8cb6283a21 460 445 2012-01-23T14:09:29Z JoergHoletschek 3 /* Adding Tables/Views with their Primary Keys */ wikitext text/x-wiki This tutorial explains in detail how to set up a datasource. == A BioCASe Data Source == Publishing a database with BioCASe means it will be made accessible through a BioCASe web service, which will be used later by the biodiversity network(s) you want to join to access to your data. For example, the GBIF Harvester will send a BioCASe Scan request to that web service to get a list of all scientific names kept in the database, and subsequently several BioCASe Search requests to load all records for these names. For each database you want to publish, you need to set up a BioCASe data source (do not mix that up with ODBC data sources on Windows machines). The resulting BioCASe web service is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source. So if you made your installation available at http://www.foobar.org/biocase during the installation process and set up a data source named Herbar, the URL of the BioCASe web service would be http://www.foobar.org/biocase/pywrapper.cgi?dsa=Herbar (data source names can be case sensitive, depending on your server’s operating system). == Creating a New Data Source == The start page of the BPS will always list the data sources currently configured, that means the web services offered by that BioCASe installation. On a fresh installation, it will be empty: [[File:instBiocaseStartpage.png|500px]] Creating a new data source can be done in the configuration tool. Choose <small>Config Tool</small> on the start page, then <small>System Administration</small>. Log in with the password you set during the installation process (if you didn’t change that yet, the default is ''ACDC''). In the configuration screen, the sections <small>Data Sources</small> and <small>Recycle Bin</small> can be used to create, remove und restore a data source: [[File:dsaCreateDatasourceEmpty.png|500px|border]] For now, we want to create a new, empty data source. Enter a name – without any spaces – in the text box and click <small>Create DSA</small> (remember that the datasource name is case sensitive on Linux machines). After a second, it will appear in the list of existing data sources with a clickable link. By doing this, you created the first BioCASe web service for your installation. Clicking <small>Remove</small> will uninstall this service again (putting it in the recycle bin, so you can restore it later). Tip: If you select <small>abcdmetadata</small> in the template drop-down list, your data source will be created with an ABCD mapping for all the metadata elements stored in the metadata table mentioned in the [[Preparation]] guide. So if you did use the templates in ''sampledb/abcdmetadata'' to create the metadata table in your database, you can use this template to spare the hassle of mapping this table and start right away with your occurrence data. == Setting up the database connection == Clicking on the data source you’ve just created will take you to the datasource configuration tool. Note that the <small>Schemas</small> section will be empty if you’ve created an empty data source or list two schemas (ABCD 1.2 and 2.06) if you created the datasource based on the <small>abcdmetadata</small> template: [[File:dsaOverview.png|450px]] The link in the <small>Database Connection</small> section will take you to the connection parameters dialog. Fill in the values for your database, press <small>Save</small> and the connection status should turn into a nice green <small>OK</small>. '''DBMS:''' Select the entry appropriate for your collection. The BPS can directly connect to database servers (MySQL, Postgres, DB2, SQL Server, and Oracle) and to Desktop-DBMS like Access and FoxPro through ODBC. ODBC is only available on Windows, so if you plan to use it, BioCASe must be installed on a Windows Server. '''Host:''' Enter the IP address of the database server. If you’re connecting through ODBC, you must leave this empty. Instead, use Windows’ ODBC Administrator to set up a System (!) datasource; the server or file name of the database will be stored in the ODBC configuration. '''Database:''' Enter the name of the ODBC datasource if you’re using ODBC, or the name of the database if you’re directly connecting to a server. '''User/Password:''' Database credentials to be used. As written in the [[Preparation]] guide, you should create a user with just read access for this. '''Encoding:''' The encoding of your database. If you’re not sure, leave the default, you can change it if you run into problems later. If you know your database is using a different encoding than latin-1 (which is typically the default for western languages), maybe UTF-8, select the appropriate entry in the drop-down list. [[File:dsaDBConnection.png|450px]] Once you’ve entered the correct values, press <small>Save</small>. Click <small>Overview</small> in the link list at the top of the page to return to the datasource configuration page. If the connection status doesn’t turn into <small>OK</small> (and doesn’t after you’ve returned to the overview page), please read the [[Debugging]] guide about how to trace the problem. == Setting up the Database Structure == Clicking on <small>Edit DB Structure</small> on the datasource overview page will show the DB structure setup. Make sure the connection status at the top of the page still is <small>OK</small>. In a first step, you will add all tables/views that hold information you want to publish with BioCASe. You can ignore all database tables storing only data you don’t plan to map to ABCD. Same holds true if you use cache tables (as explained in the [[Preparation]] guide): Only the cache tables need to be set up on the database structure page. In a second step, you will specify the foreign key references between these tables. === Adding Tables/Views with their Primary Keys === To add a table or view, just select it in the drop-down list. The name of the table will be filled in the <small>Alias</small> text box (you will use this alias later in the mapping process), feel free to change it into something that describes the contents of that tables (for example ''specimen'' for the root specimen table). Press <small>Add</small> to insert the table into the list. Before continuing to the next table, select the primary key in the list of attributes under <small>Primary Key Attribute(s)</small>. In case the data type is a string and not an integer, select <small>text</small> from the drop-down list. Press <small>Add</small> to add the primary key. If the primary key consists of several columns, you can add several primary key attributes. '''<u>Hint</u>''' It has been noted that diacritic characters in primary key columns can cause problems, so you should try to use integer or ascii text columns as primary keys. Redo these steps (adding the tables first, then the primary key/keys) for all tables you want to publish data from. Always remember to set the correct primary key column(s), even if they’re not defined as primary keys in the database. '''<u>Important Note:</u>''' Tables without a primary key cannot be used by the Provider Software and will be removed upon saving!''' === Specifying Foreign Key References === Once you’ve added all tables, you can define the relations between these tables by defining foreign keys. In order to do this, go to the <small>Foreign Key(s)</small> column of the table that holds a foreign key reference to another table, select the alias of the referenced table and press <small>Add</small>. Select the attribute that stores the foreign key and change the data type if it is not integer, then press <small>Add</small>. You must specify as many attributes for the foreign keys as the primary key of the referenced table has, otherwise the Provider Software won’t be able to create a valid join between the tables. Also, they must be in the same order and have identical data types. Redo this for all tables until you’ve specified all foreign key references. If you’ve installed the Graphviz package, the Provider Software will create a graph visualizing the current database structure. Each table will be visible as a node with the primary key(s) listed. References between tables will be represented by arrows, the foreign key(s) listed in the referring table: [[File:dsaDatabaseStructure_AlgenEngels.png|800px]] === Saving, Editing an Existing Setup === You can change aliases of tables you’ve already added to the list. To do so, go to the <small>Alias</small> text box of that table and type in a different name. Upon exiting the text box, all references to this alias will be updated in the table setup; also the graph will reflect the changes. But caution: Schema mappings use these aliases to bind schema elements to tables and columns, if you change an alias you’ve already used in a schema mapping, you will break this binding! So it is advisable to decide for meaningful alias names and keep them stable once you’ve started the mapping process. Once you’ve added a table/view to the table list, you can’t change the underlying table, since this would invalidate all the primary and foreign key attributes. If you’ve added the wrong table, remove it by using the trash can symbol next to <small>Delete Alias</small> and add the correct table/view. If you’ve finished the table setup and the graph displays the model of your database correctly, save the configuration by pressing <small>Save</small> at the bottom of the page. This is important; navigating away from the page without saving will cause your changes to be lost (even though you will be asked for confirmation)! It is advisable to save the table setup every now and then, for example once you’ve added all tables and primary keys. If you edit an existing table setup and happen to realize you’ve messed it up, you can revert to the state when you’ve pressed <small>Save</small> for the last time by using the <small>Revert</small> button at the bottom. But be careful: All unsaved changes will be lost. '''<u>Important Note:</u>''' The retrieval of tables/view/column names is currently only supported for the following DBMS: Microsoft SQL Server, MySQL, Oracle, Access, Postgres and FoxPro. If you use a different DBMS, you will see a simple text box instead of a drop-down-list, where you’ll have to type in the name of the table or column you want to add. d058710ab56d5bb600216cedcaf7ca7431074ea5 0 29 461 366 2012-02-01T18:03:35Z JoergHoletschek 3 /* Adding Mandatory ABCD Elements */ wikitext text/x-wiki This tutorial will explain how to set up a mapping for the ABCD2.06 data schema. If you intend to map another schema (e.g. one of the ABCD extension ABCD-EFG, ABCD-DNA or HISPID), you still should take the time to read it. Large portions of the process will be identical. == Access to Biological Collection Data Schema (ABCD) == ABCD in its current version 2.06 is the data schema typically used in conjunction with BioCASe. It is a highly complex XML data schema with about 1,000 elements that is able to store almost every piece of information that can be found in a natural history specimen collection or an observation database. The complexity is due to the fact that it can be used for a wide range of collections/databases – for living and preserved specimens, observations and culture collections, for zoological, botanical, bacterial and viral collections, marine or terrestrial, for herbaria, botanic and zoological gardens. For each of these special types, ABCD features special sections where information specific for this type can be stored. Other sections of ABCD will be shared by all these types, for example for gathering site, identifier or metadata. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a list of commonly used ABCD elements on the Wiki page [[CommonABCD2Concepts|Common ABCD2 Concepts]]. == Creating an ABCD Mapping for a Data Source == If you created a data source from a template (e.g. ''abcdmetadata''), a schema mapping for ABCD will be already existent. In this case, you can directly jump to the next section. If you created an empty data source, the section <small>Schemas</small> in the data source configuration overview will be empty: [[File:abcdSchemaMappings.png|border]] In order to create a new schema mapping, select the desired schema from the list and click <small>Create</small>. For ABCD, choose <small>ABCD_2.06.xml</small> (be careful not to hit one of the ABCD extensions, they look quite similar). After clicking <small>Create</small>, you will be directed to the mapping editor, which will allow you to edit the mappings. == The Mapping Editor == The mapping editor allows you to add, remove and edit ABCD mappings. On top you’ll find some summary information, on the bottom you’ll see the structure of the ABCD mapping: [[File:abcdMappingEditorEmpty.png|border]] First thing we will do is to set the root table, which is the table that holds the records you want to publish (specimens or observations). That table must hold one record per object to be published, identified by a unique identifier (called ''UnitID''). This identifier doesn't need to be the primary key necessarily, as long as it is unique (if that sentence confuses you, ignore it and just keep in mind that the UnitID must be unique.) Use the drop down box labelled <small>Root table Alias</small> to select the root table of your data model. If your metadata table holds just one record (because all occurrences published share the same metadata) and if it is not linked to the root table by a foreign key, you can specify this table in the drop-down box <small>Static table Alias</small>. The Provider Software will use a natural join to connect the metadata record stored in this table to all occurrences published. If your metadata table is referenced by a foreign key in the root able, you must not declare the metadata table as a static table! So either join the metadata table using a foreign key OR declare it as a static table. When you’ve set the root table (and static table, if you use that), press <small>Save</small>. As with the DB structure setup, this will write your changes to the configuration files, closing the tab without saving will discard your changes! Also, in case you messed something up, you can use the <small>Revert</small> button to restore the state when you’ve saved for the last time. In the mapping editor, the link <small>Overview</small> at the top of the page will take you back to the datasource configuration overview. After creating a mapping, it will appear in the section <small>Schemas</small>. The entry will show the number of mapped elements (0 for now) and the schema namespace. Pressing the trash can symbol will move it to the trash can, which means that your web service does not support this schema anymore; if you did this by mistake, you can restore it by pressing the <small>Restore</small> button. <small>Purge</small> will delete the schema mapping permanently: [[File:abcdSchemaMappings2.png|border]] You can map several schemas for a given datasource. For example, in addition to the current ABCD2 standard, you could add the deprecated version 1.2 (even though I have no idea why you should do that) or DarwinCore. Just select the appropriate schema in the drop-down list and click <small>Create</small>. Since the namespace of a schema will be used in requests, you cannot create two mappings for the same namespace. Trying this will get you an error message. == Adding Mandatory ABCD Elements == Clicking on a schema in the datasource overview page will take you again to the mapping editor. In the lower half of the page, you will see the basic structure of an ABCD document: [[File:abcdMappingEditorEmptyBottom.png|border]] <strong>Technical/Content Contact:</strong> These trees hold information on who should be contacted for technical issues or questions concerning the published database, each with name, address, email address and telephone number. <strong>Metadata:</strong> In this sub tree you’ll find all metadata fields, for example title and description of the dataset, ownership information, date last modified, IPR statements etc. <strong>Units:</strong> This part stores the "real" data you want to publish, that means all specimen or observation data. The three elements already shown in the tree are identifiers that make up the so called "Triple ID", which uniquely identifies records within primary biodiversity networks such as BioCASe and GBIF: * ''UnitID'', which is a unique identifier for each record to be published (you can use the catalogue number, for example, or an existing barcode); * ''SourceInstitutionID'', an identifier for the institution publishing the dataset (for example BGBM); * ''SourceID'', an identifier for the dataset (e.g. “Herbarium Berolinense’’). The elements already shown in the mapping editor are printed in red because they are mandatory, which means they are required for an ABCD document. Without them being mapped (to database fields that are non-empty), the Provider Software will not be able to construct valid ABCD documents and will not publish any records. So the first elements that should be mapped are the mandatory fields. To add a mapping, click the <strong>+</strong> symbol next to the field, which will open the mapping editor dialog, where you can select the database table/column you want to map the ABCD element to: [[File:abcdMappingEditorDialog.png|border]] In this case, the element ''/DataSets/DataSet/Metadata/RevisionData/DateModified'' is mapped to the column ''source_update'' of the ''metadata'' table, which has the data type ''date''. (Remember that table/column retrieval is not supported for all DBMS, so if you’re unlucky, you might need to type in table/column name manually.) <small>Save</small> will accept your settings and close the dialog, <small>Cancel</small> will discard any changes. Instead of mapping an element to a database field, you can map it to a literal. For example, you could enter the technical and content contact information directly as literals in the mapping editor dialog (assuming they’re the same for all records). However, we strongly discourage you from doing this. For one thing, this is just bad style, since it will scatter the information published over different places: Some will be retrieved from the database, some (the literals) will be loaded from the configuration files. Changing the metadata of your service later would then require access to the database and access to the BioCASe configuration – that means the configuration tool password. Be nice to the person who will do these changes and put everything neatly in a database table! The second reason is even more important: Using literals for an element will result in this element to become not searchable, which means it cannot be used in a filter of a Search or a Scan request. Even though this behaviour is discoverable with a Capabilities request (by evaluating the ''searchable'' flag), a potential user of your web service might just not think of that, so it’s a good idea to avoid that in first place. Please use literals sparsely! In the mapping editor window, you can select several source columns and several literals, even a combination of both. The Provider Software will simply concatenate the values retrieved from the database and the literals specified (in the order they’re given). But be careful: any use of concatenation or literals will make the element not searchable. A better solution is to do such manipulations earlier, in a database view, and direct the mapping editor to a single resulting database field. Use the mapping editor window to add mappings for all eight mandatory fields: Click on the <strong>+</strong> symbol next to the element, choose table/column name and the correct data type, then press <small>Save</small>. When you’re done, the page should look similar to the following image. Press <small>Save</small> in the main window to write your changes to the configuration files. [[File:abcdMappingEditorMandatoryElements.png|border]] == Initial Testing of the Data Source == Once you’ve mapped all mandatory elements, you can do the first test of your BioCASe web service. Even though you haven’t mapped any real information yet, it is a good idea to check the service and solve any problems hitherto before continuing with the mapping. Click on <small>Test mapping!</small> to open the Query Form that can be used for this purpose: [[File:abcdQueryFormEmpty.png|border]] With the Query Form you can send a BioCASe request to your web service; it will wait for the reply of the web service and display the response document. On top of the page you can see the URL of your web service. In this case it is ''<nowiki>http://localhost/biocase/pywrapper.cgi?dsa=flora</nowiki>'', ''flora'' being the name of the datasource. Make sure this is correct for your installation! Into the text box below you can enter the request XML text. Because it would be cumbersome to enter the whole BioCASe request, you can use the links below the box to load templates. For now, click on <small>ABCD2 Search</small> to fill the box with a BioCASe Search request for documents in the ABCD2 format (the one you’ve just mapped): <syntaxhighlight> <?xml version='1.0' encoding='UTF-8'?> <request xmlns='http://www.biocase.org/schemas/protocol/1.3'> <header><type>search</type></header> <search> <requestFormat>http://www.tdwg.org/schemas/abcd/2.06</requestFormat> <responseFormat start='0' limit='10'>http://www.tdwg.org/schemas/abcd/2.06</responseFormat> <filter> <like path='/DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString'>A*</like> </filter> <count>false</count> </search> </request> </syntaxhighlight> As you can see, there is a filter set on the ''ScientificName'' element. Since we haven’t mapped this element yet, this won’t work. So let’s remove the filter (all lines between <filter> and </filter>, including the ''filter'' tags), leaving the text box like this: [[File:abcdQueryFormFilled.png|border]] Pressing <small>Submit</small> will send the request to the web service and – hopefully – display the response document with the correct ABCD records. For the mapping we’ve just created, it should look similar to the following (use the "–" symbol next to the trees TechnicalContact, ContentContact and Metadata to collapse them): [[File:abcdResponseMandatoryElements.png|border]] If you get an error, please check out the [[Debugging]] tutorial on how to debug a web service. You should continue the mapping process only after your web service returns results for the basal mapping we’ve just created. == Adding additional ABCD Elements == Once you’ve successfully managed to map the mandatory ABCD elements ("successfully" means the web service returns ABCD documents), you can go on with the mapping process. This is usually an iterative process: You add a handful of elements (for example, country, country code, locality text and coordinates for the gathering site), then you use the Query Form to see if the information ends up where it is supposed to do. This is the reason the Query Form opens up in a separate tab (or window, of you configured your web browser so): In one tab, you use the mapping editor to add new elements; after pressing <small>Save</small> you go to the Query Form tab and see what the response documents look like. Redo this until you’ve mapped all information you want to publish. In order to map new ABCD concepts, tick the check box labelled <small>Show all concepts</small> and press <small>Refresh</small>. The mapping editor will then display the whole ABCD tree, which looks admittedly a bit intimidating. In order to find the concept you’re looking for, use the search function of your browser. If you want to map the country name of the gathering site, for example, hit <small>Ctrl + F</small> (for Firefox, Microsoft IE and Opera) and type in the name of the ABCD element, namely ''country''. The browser will jump to the ''Country'' tree of the ''Gathering'' element, where you’ll find the elements for country name (''Name''), ISO country code (''ISO3166Code'') and the country name in the country’s language (''NameDerived''): [[File:abcdMappingEditorSearch.png|500px]] Continue the sequence mapping new elements/testing until you’ve mapped all information you want to publish. You can have a look at the list of recommend and commonly used ABCD elements on the [[CommonABCD2Concepts|Common ABCD2 Concepts]] page. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki]. The following page will show you a sample response ABCD document for a web service with 77 concepts mapped. Of course your documents will look different, because you will probably use different ABCD elements, but still it’s good to get an idea: [[SampleABCDDocument]]. 538dbd67a594a5d1ea430a3d0d63e2afa7e79bdf 462 461 2012-02-15T15:50:31Z JoergHoletschek 3 /* The Mapping Editor */ wikitext text/x-wiki This tutorial will explain how to set up a mapping for the ABCD2.06 data schema. If you intend to map another schema (e.g. one of the ABCD extension ABCD-EFG, ABCD-DNA or HISPID), you still should take the time to read it. Large portions of the process will be identical. == Access to Biological Collection Data Schema (ABCD) == ABCD in its current version 2.06 is the data schema typically used in conjunction with BioCASe. It is a highly complex XML data schema with about 1,000 elements that is able to store almost every piece of information that can be found in a natural history specimen collection or an observation database. The complexity is due to the fact that it can be used for a wide range of collections/databases – for living and preserved specimens, observations and culture collections, for zoological, botanical, bacterial and viral collections, marine or terrestrial, for herbaria, botanic and zoological gardens. For each of these special types, ABCD features special sections where information specific for this type can be stored. Other sections of ABCD will be shared by all these types, for example for gathering site, identifier or metadata. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a list of commonly used ABCD elements on the Wiki page [[CommonABCD2Concepts|Common ABCD2 Concepts]]. == Creating an ABCD Mapping for a Data Source == If you created a data source from a template (e.g. ''abcdmetadata''), a schema mapping for ABCD will be already existent. In this case, you can directly jump to the next section. If you created an empty data source, the section <small>Schemas</small> in the data source configuration overview will be empty: [[File:abcdSchemaMappings.png|border]] In order to create a new schema mapping, select the desired schema from the list and click <small>Create</small>. For ABCD, choose <small>ABCD_2.06.xml</small> (be careful not to hit one of the ABCD extensions, they look quite similar). After clicking <small>Create</small>, you will be directed to the mapping editor, which will allow you to edit the mappings. == The Mapping Editor == The mapping editor allows you to add, remove and edit ABCD mappings. On top you’ll find some summary information, on the bottom you’ll see the structure of the ABCD mapping: [[File:abcdMappingEditorEmpty.png|border]] First thing we will do is to set the root table, which is the table that holds the records you want to publish (specimens or observations). That table must hold one record per object to be published, identified by a unique identifier (called ''UnitID''). This identifier doesn't need to be the primary key necessarily, as long as it is unique (if that sentence confuses you, ignore it and just keep in mind that the UnitID must be unique.) Use the drop down box labelled <small>Root table Alias</small> to select the root table of your data model. If your metadata table holds just one record (because all occurrences published share the same metadata) and if it is not linked to the root table by a foreign key, you can specify this table in the drop-down box <small>Static table Alias</small>. The Provider Software will use a natural join to connect the metadata record stored in this table to all occurrences published. If your metadata table is referenced by a foreign key in the root able, you must not declare the metadata table as a static table! So either join the metadata table using a foreign key OR declare it as a static table. '''Note:''' If you're using Access, you cannot use a static table, because Access doesn't support that. So when using Access, you always have to reference the metadata table with a foreign key. When you’ve set the root table (and static table, if you use that), press <small>Save</small>. As with the DB structure setup, this will write your changes to the configuration files, closing the tab without saving will discard your changes! Also, in case you messed something up, you can use the <small>Revert</small> button to restore the state when you’ve saved for the last time. In the mapping editor, the link <small>Overview</small> at the top of the page will take you back to the datasource configuration overview. After creating a mapping, it will appear in the section <small>Schemas</small>. The entry will show the number of mapped elements (0 for now) and the schema namespace. Pressing the trash can symbol will move it to the trash can, which means that your web service does not support this schema anymore; if you did this by mistake, you can restore it by pressing the <small>Restore</small> button. <small>Purge</small> will delete the schema mapping permanently: [[File:abcdSchemaMappings2.png|border]] You can map several schemas for a given datasource. For example, in addition to the current ABCD2 standard, you could add the deprecated version 1.2 (even though I have no idea why you should do that) or DarwinCore. Just select the appropriate schema in the drop-down list and click <small>Create</small>. Since the namespace of a schema will be used in requests, you cannot create two mappings for the same namespace. Trying this will get you an error message. == Adding Mandatory ABCD Elements == Clicking on a schema in the datasource overview page will take you again to the mapping editor. In the lower half of the page, you will see the basic structure of an ABCD document: [[File:abcdMappingEditorEmptyBottom.png|border]] <strong>Technical/Content Contact:</strong> These trees hold information on who should be contacted for technical issues or questions concerning the published database, each with name, address, email address and telephone number. <strong>Metadata:</strong> In this sub tree you’ll find all metadata fields, for example title and description of the dataset, ownership information, date last modified, IPR statements etc. <strong>Units:</strong> This part stores the "real" data you want to publish, that means all specimen or observation data. The three elements already shown in the tree are identifiers that make up the so called "Triple ID", which uniquely identifies records within primary biodiversity networks such as BioCASe and GBIF: * ''UnitID'', which is a unique identifier for each record to be published (you can use the catalogue number, for example, or an existing barcode); * ''SourceInstitutionID'', an identifier for the institution publishing the dataset (for example BGBM); * ''SourceID'', an identifier for the dataset (e.g. “Herbarium Berolinense’’). The elements already shown in the mapping editor are printed in red because they are mandatory, which means they are required for an ABCD document. Without them being mapped (to database fields that are non-empty), the Provider Software will not be able to construct valid ABCD documents and will not publish any records. So the first elements that should be mapped are the mandatory fields. To add a mapping, click the <strong>+</strong> symbol next to the field, which will open the mapping editor dialog, where you can select the database table/column you want to map the ABCD element to: [[File:abcdMappingEditorDialog.png|border]] In this case, the element ''/DataSets/DataSet/Metadata/RevisionData/DateModified'' is mapped to the column ''source_update'' of the ''metadata'' table, which has the data type ''date''. (Remember that table/column retrieval is not supported for all DBMS, so if you’re unlucky, you might need to type in table/column name manually.) <small>Save</small> will accept your settings and close the dialog, <small>Cancel</small> will discard any changes. Instead of mapping an element to a database field, you can map it to a literal. For example, you could enter the technical and content contact information directly as literals in the mapping editor dialog (assuming they’re the same for all records). However, we strongly discourage you from doing this. For one thing, this is just bad style, since it will scatter the information published over different places: Some will be retrieved from the database, some (the literals) will be loaded from the configuration files. Changing the metadata of your service later would then require access to the database and access to the BioCASe configuration – that means the configuration tool password. Be nice to the person who will do these changes and put everything neatly in a database table! The second reason is even more important: Using literals for an element will result in this element to become not searchable, which means it cannot be used in a filter of a Search or a Scan request. Even though this behaviour is discoverable with a Capabilities request (by evaluating the ''searchable'' flag), a potential user of your web service might just not think of that, so it’s a good idea to avoid that in first place. Please use literals sparsely! In the mapping editor window, you can select several source columns and several literals, even a combination of both. The Provider Software will simply concatenate the values retrieved from the database and the literals specified (in the order they’re given). But be careful: any use of concatenation or literals will make the element not searchable. A better solution is to do such manipulations earlier, in a database view, and direct the mapping editor to a single resulting database field. Use the mapping editor window to add mappings for all eight mandatory fields: Click on the <strong>+</strong> symbol next to the element, choose table/column name and the correct data type, then press <small>Save</small>. When you’re done, the page should look similar to the following image. Press <small>Save</small> in the main window to write your changes to the configuration files. [[File:abcdMappingEditorMandatoryElements.png|border]] == Initial Testing of the Data Source == Once you’ve mapped all mandatory elements, you can do the first test of your BioCASe web service. Even though you haven’t mapped any real information yet, it is a good idea to check the service and solve any problems hitherto before continuing with the mapping. Click on <small>Test mapping!</small> to open the Query Form that can be used for this purpose: [[File:abcdQueryFormEmpty.png|border]] With the Query Form you can send a BioCASe request to your web service; it will wait for the reply of the web service and display the response document. On top of the page you can see the URL of your web service. In this case it is ''<nowiki>http://localhost/biocase/pywrapper.cgi?dsa=flora</nowiki>'', ''flora'' being the name of the datasource. Make sure this is correct for your installation! Into the text box below you can enter the request XML text. Because it would be cumbersome to enter the whole BioCASe request, you can use the links below the box to load templates. For now, click on <small>ABCD2 Search</small> to fill the box with a BioCASe Search request for documents in the ABCD2 format (the one you’ve just mapped): <syntaxhighlight> <?xml version='1.0' encoding='UTF-8'?> <request xmlns='http://www.biocase.org/schemas/protocol/1.3'> <header><type>search</type></header> <search> <requestFormat>http://www.tdwg.org/schemas/abcd/2.06</requestFormat> <responseFormat start='0' limit='10'>http://www.tdwg.org/schemas/abcd/2.06</responseFormat> <filter> <like path='/DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString'>A*</like> </filter> <count>false</count> </search> </request> </syntaxhighlight> As you can see, there is a filter set on the ''ScientificName'' element. Since we haven’t mapped this element yet, this won’t work. So let’s remove the filter (all lines between <filter> and </filter>, including the ''filter'' tags), leaving the text box like this: [[File:abcdQueryFormFilled.png|border]] Pressing <small>Submit</small> will send the request to the web service and – hopefully – display the response document with the correct ABCD records. For the mapping we’ve just created, it should look similar to the following (use the "–" symbol next to the trees TechnicalContact, ContentContact and Metadata to collapse them): [[File:abcdResponseMandatoryElements.png|border]] If you get an error, please check out the [[Debugging]] tutorial on how to debug a web service. You should continue the mapping process only after your web service returns results for the basal mapping we’ve just created. == Adding additional ABCD Elements == Once you’ve successfully managed to map the mandatory ABCD elements ("successfully" means the web service returns ABCD documents), you can go on with the mapping process. This is usually an iterative process: You add a handful of elements (for example, country, country code, locality text and coordinates for the gathering site), then you use the Query Form to see if the information ends up where it is supposed to do. This is the reason the Query Form opens up in a separate tab (or window, of you configured your web browser so): In one tab, you use the mapping editor to add new elements; after pressing <small>Save</small> you go to the Query Form tab and see what the response documents look like. Redo this until you’ve mapped all information you want to publish. In order to map new ABCD concepts, tick the check box labelled <small>Show all concepts</small> and press <small>Refresh</small>. The mapping editor will then display the whole ABCD tree, which looks admittedly a bit intimidating. In order to find the concept you’re looking for, use the search function of your browser. If you want to map the country name of the gathering site, for example, hit <small>Ctrl + F</small> (for Firefox, Microsoft IE and Opera) and type in the name of the ABCD element, namely ''country''. The browser will jump to the ''Country'' tree of the ''Gathering'' element, where you’ll find the elements for country name (''Name''), ISO country code (''ISO3166Code'') and the country name in the country’s language (''NameDerived''): [[File:abcdMappingEditorSearch.png|500px]] Continue the sequence mapping new elements/testing until you’ve mapped all information you want to publish. You can have a look at the list of recommend and commonly used ABCD elements on the [[CommonABCD2Concepts|Common ABCD2 Concepts]] page. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki]. The following page will show you a sample response ABCD document for a web service with 77 concepts mapped. Of course your documents will look different, because you will probably use different ABCD elements, but still it’s good to get an idea: [[SampleABCDDocument]]. fded962cffd9c05d77e91db6fe66c81b84caf732 0 93 465 2012-03-29T16:43:34Z JoergHoletschek 3 Created page with "== What is the BioCASe Provider Software? == The BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information net..." wikitext text/x-wiki == What is the BioCASe Provider Software? == The BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. The Provider Software is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. The Provider Software is a suite of tools that need to be installed on the data provider’s web server. The core component is the PyWrapper, an XML/CGI database interface written in Python that allows a standardized access to a variety of database management systems and arbitrarily structured databases. Grouped around this are a number of tools for configuring, testing and debugging a BioCASe installation or web service. Even though BioCASe can be used for any conceptual XML data schema, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the [http://www.biocase.org BioCASe network] and the [http://www.gbif.org Global Biodiversity Information Facility]. == Requirements for Using the BPS == Before using the BioCASe Provider Software, three requirements need to be checked: * For one thing, the data to be published needs to be stored in an SQL compliant relational database management system (DBMS), with the data model being well documented or at least knowledge about the data model existing on someone’s head. Non-SQL DBMS are not supported. Currently, the BioCASe Provider Software can connect to the following databases: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * For installing BioCASe, you need to have a web server (Apache or Microsoft IIS) that is permanently connected to the Internet. This can be an institutional server or a hosting service (even though it’s usually a pain to get privileges to install software for the latter). Dial-up connections won’t work. * For the web server in question, you must have the possibility to install Python (if it is not installed yet) and Python packages. Even though this can be done by the IT staff, you should make sure that there are no institutional policies colliding. == Steps == Once you’ve made sure the BioCASe Provider Software is the right choice for you ([[Special:Contact|contact the BioCASe team]] if you feel unsafe with your decision), you need to do the following steps: # Think about which information you want to publish to the network. This involves settling the question of ownership of the data and the terms of use, license and usage restrictions for the published data. Also you should consider which information should '''not''' be published because they might affect endangered species (options are to block certain occurrences from publication or to blur the exact locality information). # Prepare the database for publication as described in the [[Preparation]] tutorial. This includes the decision on whether to publish the live database or a snapshot, the creation of a table for metadata and, in certain cases, some data transformation. # Follow the [[Installation]] Tutorial to install the BioCASe Provider Software. # Set up a BioCASe Datasource and connect it to your database, then configure the database structure as described in the [[DatasourceSetup]] tutorial. # Follow the [[ABCD2Mapping]] tutorial to create a mapping for the schema you want to use. # Test and debug your BioCASe web service. Read the [[Debugging]] tutorial to learn how to do this. # Once you’re done with setting up and configuring the BioCASe Provider Software, you can register the BioCASe web service with the information network you want to publish your data to, for example with [http://www.gbif.org GBIF]. We strongly recommend to [[Special:Contact|contact the BioCASe team]] for checking your web service first as we might be able to foresee problems that are more difficult to resolve after registration. == Support == In case you run into difficulties during the installation and setup, you should * have a look at the [[FAQ]] which might list the solution for your question, * ask other BioCASe data providers for help, * get in contact with the [[Special:Contact|BioCASe team]]. Please describe exactly you environment (operating system, DBMS used), the steps that cause the problem and any error messages and hints you get. Screenshot are a big plus here. e3b2ca7c93155c3a8e8f452167008264b1f63bfa 0 94 466 2012-03-29T16:44:24Z JoergHoletschek 3 Created page with "This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please re..." wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can a pain; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a Postgres and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] b0134e79c2a2e9446545eab65c8cc61da2914398 467 466 2012-03-29T17:33:13Z FranckTheeten 15 wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can a pain; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a Postgres and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] Installation From Software Provider BioCASE Aller à: Navigation, rechercher Installation Cette partie du wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé BPS). Cela devrait prendre entre 15 et 60 minutes. Si vous êtes un débutant, s'il vous plaît lire le Guide du Débutant en premier lieu. Contenu * 1 Prérequis * 2 Installation de Apache / IIS et Python * 3 Téléchargement du logiciel fournisseur de données o 3.1 Téléchargement du fichier d'archive o 3,2 téléchargement à partir du serveur Subversion * 4 Exécution du script d'installation * 5 Lier BioCASe et serveur Web o 5.1 Apache o Internet Information Server 5.2 * 6 Installation de paquets Python supplémentaires * 7 Modification du mot de passe par défaut * 8 Vérification de l'installation Prérequis Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: 1. Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. 2. Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit soit être en dehors de tout pare-feu institutionnel ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de stasifaire cette condition si vous installez BioCASe a des fins de tests ou de travail individuel sur votre propre ordinateur]. 3. Une connexion directe peut être établie à partir de cette machine à la base de données à publier - soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système de l'aide, lisez la FAQ ou prenez contact avec l'équipe BioCASe (support@biocase.org et /ou du projet CABIN). Installation de Apache / IIS et Python Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallé. Il en va de même pour MacOS X, mais vous devriez jeter un oeil aux discussions et soruces de documentations accessible sur web pour en savoir plus sur les emplacements de fichiers. Même provider fonctionne avec Apache et Internet Information Server (IIS), nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous voudrez peut-être rester sur cette plateforme. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite Python version 2.5 (et autres sous-versions) afin de fonctionner. Même s’il fontionne également avec les versions 2.6 et 2.7, la version recommandée est de 2,5 Python. Pour les versions ultérieures, les librairies nécessaire pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin , trouver le paquet requis pour votre système gestionnaire de bases de données et vérifier la disponibilité de nouvelles versions de Python sur le site Web respectif (par exemple, la bibliothèque 4SuiteXML requis pour l'exécution l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous faites l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier Program Files par défaut, car installé à cet endroit le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors du changement de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Quand vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. Téléchargement du logiciel fournisseur Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir de notre dépôt Subversion. La première option est la plus facile si vous n'êtes pas familier avec Subversion; avec la deuxième option, il est plus facile pour vous de mettre à jour votre installation ultérieurement. Donc, si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mis à jour, vous devriez opter plutôt pour le dépôt subversion. Téléchargement du fichier d'archive Il suffit de télécharger l'archive à partir de la section Téléchargement du site BioCASe et décompresser le contenu du dossier de destination. Téléchargement à partir du dépôt Subversion Si vous utilisez Windows, vous devez installer un client Subversion premier (Tortoise , qui est livré avec une jolie interface très commode). Linux et MacOS contiennent généralement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La lise à jour de votre installation à la dernière version BPS est alors facile et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 (dans le cas de la version 2.6.0). Pour télécharger le BPS avec le client Subversion en ligne de commande (préinstallé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase Vous vous connecterez au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire l'BioCASe.. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez « Checkout». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: Exécution du script d'installation Le script d'installation lancera les fichiers de configuration avec les valeurs que vous fournissez et liera les fichiers de script à votre installation de Python (vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement des répertoires d’Apache, Python ou du logiciel lui-même sur votre ordinateur). Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (ou «invite de commande" dans Windows, à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter »). Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS (par exemple « cd D://biocaseprovider ») et tapez: python setup.py Si la commande ne marche pas , vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python. Exemple (sous Windows et DOS, en supposant que vous avez installé python dans sa version 2.5 et dans le répertoire C:\Python25. Adaptez les valeurs en fonction de la situation sur votre ordinateur) C:\Python25\setup.py setup.py Assurez-vous que la version de Python donnée dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de l’intepréteur comme décrit ci-dessus): InstSetupScriptStart.png Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur Entrée. Domaine Webserver: Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, http://www.moninstitution.org. Vous pouvez conserver le localhost par défaut tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) plus tard. URL de base: Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par xemple http://moinstitution.org/monbiocase/ ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à www.myinstitution.org/biocase. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! 6b52206d01aa49228ceece24db88ad459aa50e25 468 467 2012-03-29T17:39:34Z FranckTheeten 15 wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can a pain; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a Postgres and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] Installation From Software Provider BioCASE Aller à: Navigation, rechercher Installation Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes. Si vous êtes un débutant, veuillez lire le Guide du Débutant en premier lieu. Contenu * 1 Prérequis * 2 Installation de Apache / IIS et Python * 3 Téléchargement du logiciel fournisseur de données o 3.1 Téléchargement du fichier d'archive o 3,2 téléchargement à partir du serveur Subversion * 4 Exécution du script d'installation * 5 Lier BioCASe et serveur Web o 5.1 Apache o Internet Information Server 5.2 * 6 Installation de paquets Python supplémentaires * 7 Modification du mot de passe par défaut * 8 Vérification de l'installation Prérequis Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: 1. Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. 2. Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit soit être en dehors de tout pare-feu institutionnel ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de stasifaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. 3. Une connexion directe peut être établie à partir de cette machine à la base de données à publier - soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système de l'aide, lisez la FAQ ou prenez contact avec l'équipe BioCASe. Installation de Apache / IIS et Python Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez jeter un oeil aux discussions et documentations accessible sur web pour en savoir plus sur les emplacements de fichiers dans le système. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous voudrez peut-être rester sur cette plateforme. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fontionner également avec les versions 2.6 et 2.7, la version recommandée est de 2.5 Python (ou la dernière sous-version stable de la branche 2.5). Pour les versions ultérieures, les librairies nécessaire pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin pour vous procurer le paquet requis pour l'installation du driver de votre base de données et vérifier la disponibilité de nouvelles versions de Python sur le site Web respectif (par exemple, la bibliothèque 4SuiteXML uitlisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous faites l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier Program Files par défaut, car installé à cet endroit le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors du changement de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Quand vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. Téléchargement du logiciel fournisseur Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir de notre dépôt Subversion. La première option est la plus facile si vous n'êtes pas familier avec Subversion; avec la deuxième option, il est plus facile pour vous de mettre à jour votre installation ultérieurement. Donc, si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mis à jour, vous devriez opter plutôt pour le dépôt subversion. Téléchargement du fichier d'archive Il suffit de télécharger l'archive à partir de la section Téléchargement du site BioCASe et décompresser le contenu du dossier de destination. Téléchargement à partir du dépôt Subversion Si vous utilisez Windows, vous devez installer un client Subversion premier (Tortoise , qui est livré avec une jolie interface très commode). Linux et MacOS contiennent généralement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La lise à jour de votre installation à la dernière version BPS est alors facile et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 (dans le cas de la version 2.6.0). Pour télécharger le BPS avec le client Subversion en ligne de commande (préinstallé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase Vous vous connecterez au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire l'BioCASe.. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez « Checkout». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: Exécution du script d'installation Le script d'installation lancera les fichiers de configuration avec les valeurs que vous fournissez et liera les fichiers de script à votre installation de Python (vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement des répertoires d’Apache, Python ou du logiciel lui-même sur votre ordinateur). Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (ou «invite de commande" dans Windows, à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter »). Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS (par exemple « cd D://biocaseprovider ») et tapez: python setup.py Si la commande ne marche pas , vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python. Exemple (sous Windows et DOS, en supposant que vous avez installé python dans sa version 2.5 et dans le répertoire C:\Python25. Adaptez les valeurs en fonction de la situation sur votre ordinateur) C:\Python25\setup.py setup.py Assurez-vous que la version de Python donnée dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de l’intepréteur comme décrit ci-dessus): InstSetupScriptStart.png Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur Entrée. Domaine Webserver: Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, http://www.moninstitution.org. Vous pouvez conserver le localhost par défaut tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) plus tard. URL de base: Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par xemple http://moinstitution.org/monbiocase/ ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à www.myinstitution.org/biocase. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! 1b208bebb467165b6eb5b755ca40655e9b3641cb 0 23 469 458 2012-04-03T11:03:45Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == ===Upcoming=== * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for XML archives when using IIS (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool d083906f38af552246a3c865c603e77ceb2d97d9 477 469 2012-04-04T10:48:43Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == ===Upcoming=== * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for XML archives when using IIS (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 8cdc996fd76939aa85b0c86dca4d0b048b44640a 478 477 2012-04-04T13:57:38Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == ===Upcoming=== * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources * Setup script now fixes write permissions for archive folders (working/download directory) * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for XML archives when using IIS (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool d551a419e51318fe0558442a449f0328ed990f51 479 478 2012-04-04T14:15:33Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == ===Upcoming=== * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for XML archives when using IIS (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 45bacd377e6dcde99e7b99516baaecae1887c270 480 479 2012-04-04T14:33:26Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == ===Upcoming=== * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for XML archives when using IIS (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 11b72d2ea4909f5e7b6583d16be61c0ef0225fb2 487 480 2012-04-05T14:48:04Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == ===Version 3.0.1 [2012-04-05]=== * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for XML archives when using IIS (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 92e83ad2f79bb1729c93323d9cc663e78a221d2c 499 487 2012-04-10T14:29:43Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Fix: XML declaration added to documents in XML archives ===Version 3.0.1 [2012-04-05]=== * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for XML archives when using IIS (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 388512c8c3731183949cae281927041da7d59aa5 0 6 470 370 2012-04-04T09:33:18Z JoergHoletschek 3 /* Internet Information Server */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a Postgres and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] 494357a1fe31d7c0e4d42c066a72d04ca9daed21 472 470 2012-04-04T09:34:26Z JoergHoletschek 3 /* Internet Information Server */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|300px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a Postgres and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] 243bc1d61bb53e328b5924d1ab370746609560e6 473 472 2012-04-04T09:35:18Z JoergHoletschek 3 /* Linking BioCASe and Your Web Server */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|300px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html === Testing the Link === Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a Postgres and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] 6a218dd269094a42e7916220d77874f540650ad4 474 473 2012-04-04T09:35:42Z JoergHoletschek 3 /* Internet Information Server */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html === Testing the Link === Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a Postgres and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] 88ae6175aae57a4d9f9d5208eea1ace47e18cdc5 6 95 471 2012-04-04T09:33:38Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 0 52 475 459 2012-04-04T09:51:20Z JoergHoletschek 3 /* Installation */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an "Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request" message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I'm using Internet Information Server, and after creating the Virtual Directory for BioCASe, I get a "502.2 - Bad Gateway. The specified CGI application misbehaved by not returning a complete set of HTTP headers" error.=== This happens when you didn't correctly associate .cgi and .py files with Python. Make sure you've linked both files with Python and that the path contains the placeholders ''%s %s'' (see [[Installation#Internet_Information_Server|Linking IIS]]). ===I cannot find the installation package of the database connectivity module for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later of psycopg2. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation to the ''config'' folder of your new installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. You can then edit new installation and test the web services. ===I've updated my installation from a version prior to 2.6. Now, when I enter the DB structure, some of the table and column names are gone, the same happens in the mapping editor.=== Starting with version 2.6, BioCASe retrieves table and column names from the database for some database management systems (namely MySQL, SQL Server, Postgres, Foxpro, Access and Oracle). In older versions, you had to enter table/column names manually, and when you did the setup with such an early version, you probably didn't care for upper/lower cases. Since some DB management systems are case-sensitive, BioCASe can't ignore the differences and won't find these technically incorrect names in the lists retrieved from the database. The respective drop down boxes will therefore be empty. If that happens, just select the correct names from the drop down lists. ===When I try to create an archive, I get a "Permission denied" error. Why is that?=== For creating archives, you need to grant write permissions for the folders ''archive'' and ''www/downloads'' of your BioCASe installation to Python. You can check if permissions are set correctly in the test libs page (Start --> Utilities --> Test Libs). == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other only by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===I've mapped ABCD2. When testing the web service, I get a negative number for "RecordsDropped" in the response document. What does that mean?=== Same problem as above - it means you have an error in your mapping. Most probably you've mapped a non-repeatable ABCD concept to a table/column in your database that holds several entries per unit (occurrence) record. The BioCASe Provider Software will duplicate the respective unit records to be able to create valid ABCD documents - which is indicated by a negative number of "records dropped". Make sure the cardinality of your database relations match the structure (in respects of element repeatability) of ABCD to avoid this problem. ===I'm using pymssql to connect to a SQL Server database. Char(n) and Varchar(n) strings get truncated to 255 characters - what's wrong?=== This is known limitation of TDS protocol used by pymssql. Just change the data type for columns storing long string to TEXT or create a view that converts Char/Varchar columns to TEXT columns on the fly to work around this issue. ===I'm using pymssql to connect to a SQL Server database, BioCASe is running on a Linux distribution. When I do a Search or Scan request, I don't get any results (0 hits), even though there are definitely records in the database.=== That problem occurs on old versions of pymssql (1.0.2 or earlier) on Linux distributions. Update to version 2.0 or later to solve this. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. == Trouble Shooting == ===I’ve forgotten the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. c88e01946999d2617dc082232020ead02a8499ae 476 475 2012-04-04T09:52:03Z JoergHoletschek 3 /* Installation */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I'm using Internet Information Server, and after creating the Virtual Directory for BioCASe, I get a “502.2 - Bad Gateway. The specified CGI application misbehaved by not returning a complete set of HTTP headers” error.=== This happens when you didn't correctly associate .cgi and .py files with Python. Make sure you've linked both files with Python and that the path contains the placeholders ''%s %s'' (see [[Installation#Internet_Information_Server|Linking IIS]]). ===I cannot find the installation package of the database connectivity module for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later of psycopg2. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it. Download and install the new version as described in the Installation guide (lucky enough, you can skip the installation of Apache, Python and any DB packages). Before pointing your web server to the new installation, manually copy the configuration files located in the ''config'' folder of your old installation to the ''config'' folder of your new installation: ''config.ini'' for the system settings and all folders in the ''datasources'' directory. You can then edit new installation and test the web services. ===I've updated my installation from a version prior to 2.6. Now, when I enter the DB structure, some of the table and column names are gone, the same happens in the mapping editor.=== Starting with version 2.6, BioCASe retrieves table and column names from the database for some database management systems (namely MySQL, SQL Server, Postgres, Foxpro, Access and Oracle). In older versions, you had to enter table/column names manually, and when you did the setup with such an early version, you probably didn't care for upper/lower cases. Since some DB management systems are case-sensitive, BioCASe can't ignore the differences and won't find these technically incorrect names in the lists retrieved from the database. The respective drop down boxes will therefore be empty. If that happens, just select the correct names from the drop down lists. ===When I try to create an archive, I get a "Permission denied" error. Why is that?=== For creating archives, you need to grant write permissions for the folders ''archive'' and ''www/downloads'' of your BioCASe installation to Python. You can check if permissions are set correctly in the test libs page (Start --> Utilities --> Test Libs). == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other only by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===I've mapped ABCD2. When testing the web service, I get a negative number for "RecordsDropped" in the response document. What does that mean?=== Same problem as above - it means you have an error in your mapping. Most probably you've mapped a non-repeatable ABCD concept to a table/column in your database that holds several entries per unit (occurrence) record. The BioCASe Provider Software will duplicate the respective unit records to be able to create valid ABCD documents - which is indicated by a negative number of "records dropped". Make sure the cardinality of your database relations match the structure (in respects of element repeatability) of ABCD to avoid this problem. ===I'm using pymssql to connect to a SQL Server database. Char(n) and Varchar(n) strings get truncated to 255 characters - what's wrong?=== This is known limitation of TDS protocol used by pymssql. Just change the data type for columns storing long string to TEXT or create a view that converts Char/Varchar columns to TEXT columns on the fly to work around this issue. ===I'm using pymssql to connect to a SQL Server database, BioCASe is running on a Linux distribution. When I do a Search or Scan request, I don't get any results (0 hits), even though there are definitely records in the database.=== That problem occurs on old versions of pymssql (1.0.2 or earlier) on Linux distributions. Update to version 2.0 or later to solve this. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. == Trouble Shooting == ===I’ve forgotten the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. b9ff08864f14d57e69d183c23e2d9b5d45515f1d 0 17 481 249 2012-04-05T09:22:45Z JoergHoletschek 3 /* Repeatable Elements in ABCD */ wikitext text/x-wiki Before you start publishing your database with BioCASe, you need to consider several issues. In some cases, some preparation of your database for the publication might be necessary. You should read this page of the wiki careful before you start configuring the Provider Software for a given database. == Live-DB versus Snapshot == The easiest way is to connect BioCASe to the same database you’re working with. This would result in any changes you do being visible on the BioCASe access point right away – which is an idea you might not like. Some curators prefer to publish defined versions of their database, for example after they’ve finished reviewing all specimens of a certain genus. If you prefer this, you need to create a second database that holds defined snapshots of your working database and that will be published instead. Snapshots could be generated in a regular interval (e.g., each Friday evening) or upon request. Please contact your IT guys for more help on this. == Access Control == The BioCASe Provider Software will only publish your data to biodiversity networks, meaning it will only read from the database and never do any changes. For security reasons it’s a good idea to create a user for BioCASe that has only read privileges on the database – more precisely on the tables you want to publish. Since the configuration tool will try to load a list of tables and columns from the database to allow easier setup, this user should also be allowed to load meta-information (<small>SHOW TABLES, SHOW COLUMNS</small>). == Metadata == Metadata describe your dataset as a whole - title, description, curator’s name, ownership, date of last modification, terms of use etc. BioCASe can publish a lot of metadata – and you should take the time to provide as much information as you can, since this will increase the usability of your published data. For storing metadata, please create an extra table in your database and fill it in. If you go to the BioCASe folder of your installation, you’ll find the template for this table in ''sampledb/abcdmetadata''. Either use the SQL script to create it in your database, or import it from the Access database (''metadata.mdb''). Per default, all records of your database will make up one dataset, being described by one set of metadata. In case your database holds records of several datasets, you can also group these into several datasets in ABCD by providing different sets of metadata. So when you realize during gathering of meta-information that they’re not the same for all records, this is an indicator that your database must be grouped into several datasets. If all records to be published share the same metadata, the metadata table needs to have just one row. If they don’t, it needs to hold several rows, one set of metadata for each dataset, identified by a unique primary key (1, 2, …). In order to associate each record of your database with the correct set of metadata, you will need to add a foreign key into the base record table that links to the respective entry in the metadata table. Ask your IT guys if you need help on that. == Controlled Denormalisation == As you will see during configuration, the Provider Software is able to join together the different tables of your database’s data model in order to retrieve all information to be published from the database. This works good if the number of tables is within reasonable limits, that is up to approximately 10-15. Data models used by off-the-shelf collection management system are usually highly normalised and use far more tables. Specify’s data model, for example, has about 70 tables. Configuring such a complex model in BioCASe will be a terrible pain; moreover, in case your database holds a huge number of records, the time required by the Provider Software for answering search request will become too long. Therefore it is advisable in such cases to add some controlled denormalisation to your database, that is to export the data to be published into several cache tables. These tables could be stored in the same database as the source tables or separate in an extra cache database, maybe even on a different server. They could be created in a regular interval (e.g. each night) or upon request by SQL scripts. The figure below shows this denormalisation process for the data model of Specify. On the left side, you see some of Specify’s source tables (only the ones starting with the letters A, B, C and D). With an SQL script they will be reduced to 7 cache tables, one of them being abcd_object. These tables hold only the columns that will be published, drawn from up to 15 source tables. [[image:prepDenormalisation.png|750px]] == Repeatable Elements in ABCD == One of the main features of ABCD (the schema you will probably use to publish your data) is repeatable elements. For example, ABCD can store multiple identifications or several URLs for multimedia objects attached to a specimen record. All these URLs will be stored in the same ABCD element – namely ''MultimediaObject/FileURI''. You can tell repeatable elements in ABCD by the existence of a container element. For ''MultimediaObject'', for example, there is a container element ''MultimediaObjects'' one level above which holds the (potentially) several multimedia objects: Another example is the higher taxonomy (all taxa above genus level). Typically these are stored in different columns in databases (see figure below). In ABCD, they will all end up in the same element, ''HigherTaxon/Name''. In order to be able to map the different ranks stored in columns to the same ABCD concept, we need to pivot these columns into rows – each rank kept in a column needs to be turned into a row, with two columns storing taxon name and rank, one column for linking to the according specimen record, plus one newly created primary key: [[File:prepPivoting.png]] Such a transformation can be easily done with a database view. Views do exactly what the name suggests – they provide a different view on the data. They can be used like regular tables, but they merely represent a different representation of another table. Updates of the underlying tables (or insert/deletions) will also affect views created on these tables, since the data is stored only once in the database. Let’s assume you have the higher taxonomy stored in the specimen table like this: [[File:prepEditTable.png]] The SQL statement <pre> CREATE VIEW [dbo].[vwHigherTaxa] AS SELECT 'k_' + [EDIT_ATBI_RecordID] AS id, [EDIT_ATBI_RecordID] AS unit_id, [kingdom] AS name, 'kingdom' AS rank FROM unit_data WHERE [kingdom] IS NOT NULL UNION ALL SELECT 'p_' + [EDIT_ATBI_RecordID], [EDIT_ATBI_RecordID], [phylum], 'phylum'‚ FROM unit_data WHERE [phylum] IS NOT NULL UNION ALL ... </pre> will create a view that meets the requirements described above. Remember that even though it looks like a table, it’s just another representation of the data stored in the table unit_data: [[File:prepEditView.png]] Such pivoting must be done in all cases when several columns need to be mapped to the same (repeatable) ABCD element. If it’s just one value to be mapped, there is no need for that. Commonly used ABCD elements that are repeatable and might ask for such preparatory transformations are * Identification * HigherTaxon * GatheringSite/NamedArea * Metadata/Scope/GeoecologicalTerms * Metadata/Scope/TaxonomicTerms * MultimediaObjects * MeasurementsOrFacts If you have problems preparing your database for publication, please [[Special:Contact|contact the BioCASe team]]. f7c6146fdb9340de926029e8bacd41303ad3d0af 482 481 2012-04-05T09:25:53Z JoergHoletschek 3 /* Repeatable Elements in ABCD */ wikitext text/x-wiki Before you start publishing your database with BioCASe, you need to consider several issues. In some cases, some preparation of your database for the publication might be necessary. You should read this page of the wiki careful before you start configuring the Provider Software for a given database. == Live-DB versus Snapshot == The easiest way is to connect BioCASe to the same database you’re working with. This would result in any changes you do being visible on the BioCASe access point right away – which is an idea you might not like. Some curators prefer to publish defined versions of their database, for example after they’ve finished reviewing all specimens of a certain genus. If you prefer this, you need to create a second database that holds defined snapshots of your working database and that will be published instead. Snapshots could be generated in a regular interval (e.g., each Friday evening) or upon request. Please contact your IT guys for more help on this. == Access Control == The BioCASe Provider Software will only publish your data to biodiversity networks, meaning it will only read from the database and never do any changes. For security reasons it’s a good idea to create a user for BioCASe that has only read privileges on the database – more precisely on the tables you want to publish. Since the configuration tool will try to load a list of tables and columns from the database to allow easier setup, this user should also be allowed to load meta-information (<small>SHOW TABLES, SHOW COLUMNS</small>). == Metadata == Metadata describe your dataset as a whole - title, description, curator’s name, ownership, date of last modification, terms of use etc. BioCASe can publish a lot of metadata – and you should take the time to provide as much information as you can, since this will increase the usability of your published data. For storing metadata, please create an extra table in your database and fill it in. If you go to the BioCASe folder of your installation, you’ll find the template for this table in ''sampledb/abcdmetadata''. Either use the SQL script to create it in your database, or import it from the Access database (''metadata.mdb''). Per default, all records of your database will make up one dataset, being described by one set of metadata. In case your database holds records of several datasets, you can also group these into several datasets in ABCD by providing different sets of metadata. So when you realize during gathering of meta-information that they’re not the same for all records, this is an indicator that your database must be grouped into several datasets. If all records to be published share the same metadata, the metadata table needs to have just one row. If they don’t, it needs to hold several rows, one set of metadata for each dataset, identified by a unique primary key (1, 2, …). In order to associate each record of your database with the correct set of metadata, you will need to add a foreign key into the base record table that links to the respective entry in the metadata table. Ask your IT guys if you need help on that. == Controlled Denormalisation == As you will see during configuration, the Provider Software is able to join together the different tables of your database’s data model in order to retrieve all information to be published from the database. This works good if the number of tables is within reasonable limits, that is up to approximately 10-15. Data models used by off-the-shelf collection management system are usually highly normalised and use far more tables. Specify’s data model, for example, has about 70 tables. Configuring such a complex model in BioCASe will be a terrible pain; moreover, in case your database holds a huge number of records, the time required by the Provider Software for answering search request will become too long. Therefore it is advisable in such cases to add some controlled denormalisation to your database, that is to export the data to be published into several cache tables. These tables could be stored in the same database as the source tables or separate in an extra cache database, maybe even on a different server. They could be created in a regular interval (e.g. each night) or upon request by SQL scripts. The figure below shows this denormalisation process for the data model of Specify. On the left side, you see some of Specify’s source tables (only the ones starting with the letters A, B, C and D). With an SQL script they will be reduced to 7 cache tables, one of them being abcd_object. These tables hold only the columns that will be published, drawn from up to 15 source tables. [[image:prepDenormalisation.png|750px]] == Repeatable Elements in ABCD == One of the main features of ABCD (the schema you will probably use to publish your data) is repeatable elements. For example, ABCD can store multiple identifications or several URLs for multimedia objects attached to a specimen record. All these URLs will be stored in the same ABCD element – namely ''MultimediaObject/FileURI''. You can tell repeatable elements in ABCD by the existence of a container element. For ''MultimediaObject'', for example, there is a container element ''MultimediaObjects'' one level above which holds the (potentially) several multimedia objects: [[File:prepMultimediaObjects.png]] Another example is the higher taxonomy (all taxa above genus level). Typically these are stored in different columns in databases (see figure below). In ABCD, they will all end up in the same element, ''HigherTaxon/Name''. In order to be able to map the different ranks stored in columns to the same ABCD concept, we need to pivot these columns into rows – each rank kept in a column needs to be turned into a row, with two columns storing taxon name and rank, one column for linking to the according specimen record, plus one newly created primary key: [[File:prepPivoting.png]] Such a transformation can be easily done with a database view. Views do exactly what the name suggests – they provide a different view on the data. They can be used like regular tables, but they merely represent a different representation of another table. Updates of the underlying tables (or insert/deletions) will also affect views created on these tables, since the data is stored only once in the database. Let’s assume you have the higher taxonomy stored in the specimen table like this: [[File:prepEditTable.png]] The SQL statement <pre> CREATE VIEW [dbo].[vwHigherTaxa] AS SELECT 'k_' + [EDIT_ATBI_RecordID] AS id, [EDIT_ATBI_RecordID] AS unit_id, [kingdom] AS name, 'kingdom' AS rank FROM unit_data WHERE [kingdom] IS NOT NULL UNION ALL SELECT 'p_' + [EDIT_ATBI_RecordID], [EDIT_ATBI_RecordID], [phylum], 'phylum'‚ FROM unit_data WHERE [phylum] IS NOT NULL UNION ALL ... </pre> will create a view that meets the requirements described above. Remember that even though it looks like a table, it’s just another representation of the data stored in the table unit_data: [[File:prepEditView.png]] Such pivoting must be done in all cases when several columns need to be mapped to the same (repeatable) ABCD element. If it’s just one value to be mapped, there is no need for that. Commonly used ABCD elements that are repeatable and might ask for such preparatory transformations are * Identification * HigherTaxon * GatheringSite/NamedArea * Metadata/Scope/GeoecologicalTerms * Metadata/Scope/TaxonomicTerms * MultimediaObjects * MeasurementsOrFacts If you have problems preparing your database for publication, please [[Special:Contact|contact the BioCASe team]]. 3fd233ef04d5d34c1b9ff6d8516cd5f844f469e4 484 482 2012-04-05T09:27:14Z JoergHoletschek 3 /* Repeatable Elements in ABCD */ wikitext text/x-wiki Before you start publishing your database with BioCASe, you need to consider several issues. In some cases, some preparation of your database for the publication might be necessary. You should read this page of the wiki careful before you start configuring the Provider Software for a given database. == Live-DB versus Snapshot == The easiest way is to connect BioCASe to the same database you’re working with. This would result in any changes you do being visible on the BioCASe access point right away – which is an idea you might not like. Some curators prefer to publish defined versions of their database, for example after they’ve finished reviewing all specimens of a certain genus. If you prefer this, you need to create a second database that holds defined snapshots of your working database and that will be published instead. Snapshots could be generated in a regular interval (e.g., each Friday evening) or upon request. Please contact your IT guys for more help on this. == Access Control == The BioCASe Provider Software will only publish your data to biodiversity networks, meaning it will only read from the database and never do any changes. For security reasons it’s a good idea to create a user for BioCASe that has only read privileges on the database – more precisely on the tables you want to publish. Since the configuration tool will try to load a list of tables and columns from the database to allow easier setup, this user should also be allowed to load meta-information (<small>SHOW TABLES, SHOW COLUMNS</small>). == Metadata == Metadata describe your dataset as a whole - title, description, curator’s name, ownership, date of last modification, terms of use etc. BioCASe can publish a lot of metadata – and you should take the time to provide as much information as you can, since this will increase the usability of your published data. For storing metadata, please create an extra table in your database and fill it in. If you go to the BioCASe folder of your installation, you’ll find the template for this table in ''sampledb/abcdmetadata''. Either use the SQL script to create it in your database, or import it from the Access database (''metadata.mdb''). Per default, all records of your database will make up one dataset, being described by one set of metadata. In case your database holds records of several datasets, you can also group these into several datasets in ABCD by providing different sets of metadata. So when you realize during gathering of meta-information that they’re not the same for all records, this is an indicator that your database must be grouped into several datasets. If all records to be published share the same metadata, the metadata table needs to have just one row. If they don’t, it needs to hold several rows, one set of metadata for each dataset, identified by a unique primary key (1, 2, …). In order to associate each record of your database with the correct set of metadata, you will need to add a foreign key into the base record table that links to the respective entry in the metadata table. Ask your IT guys if you need help on that. == Controlled Denormalisation == As you will see during configuration, the Provider Software is able to join together the different tables of your database’s data model in order to retrieve all information to be published from the database. This works good if the number of tables is within reasonable limits, that is up to approximately 10-15. Data models used by off-the-shelf collection management system are usually highly normalised and use far more tables. Specify’s data model, for example, has about 70 tables. Configuring such a complex model in BioCASe will be a terrible pain; moreover, in case your database holds a huge number of records, the time required by the Provider Software for answering search request will become too long. Therefore it is advisable in such cases to add some controlled denormalisation to your database, that is to export the data to be published into several cache tables. These tables could be stored in the same database as the source tables or separate in an extra cache database, maybe even on a different server. They could be created in a regular interval (e.g. each night) or upon request by SQL scripts. The figure below shows this denormalisation process for the data model of Specify. On the left side, you see some of Specify’s source tables (only the ones starting with the letters A, B, C and D). With an SQL script they will be reduced to 7 cache tables, one of them being abcd_object. These tables hold only the columns that will be published, drawn from up to 15 source tables. [[image:prepDenormalisation.png|750px]] == Repeatable Elements in ABCD == One of the main features of ABCD (the schema you will probably use to publish your data) is repeatable elements. For example, ABCD can store multiple identifications or several URLs for multimedia objects attached to a specimen record. All these URLs will be stored in the same ABCD element – namely ''MultimediaObject/FileURI''. You can tell repeatable elements in ABCD by the existence of a container element. For ''MultimediaObject'', for example, there is a container element ''MultimediaObjects'' one level above which holds the (potentially) several multimedia objects: [[File:prepMultimediaObjects.png]] Another example is the higher taxonomy (all taxa above genus level). Typically these are stored in different columns in databases (see figure below). In ABCD, they will all end up in the same element, ''HigherTaxon/Name''. In order to be able to map the different ranks stored in columns to the same ABCD concept, we need to pivot these columns into rows – each rank kept in a column needs to be turned into a row, with two columns storing taxon name and rank, one column for linking to the according specimen record, plus one newly created primary key: [[File:prepPivoting.png]] Such a transformation can be easily done with a database view. Views do exactly what the name suggests – they provide a different view on the data. They can be used like regular tables, but they merely represent a different representation of another table. Updates of the underlying tables (or insert/deletions) will also affect views created on these tables, since the data is stored only once in the database. Let’s assume you have the higher taxonomy stored in the specimen table like this: [[File:prepEditTable.png]] The SQL statement <pre> CREATE VIEW [dbo].[vwHigherTaxa] AS SELECT 'k_' + [EDIT_ATBI_RecordID] AS id, [EDIT_ATBI_RecordID] AS unit_id, [kingdom] AS name, 'kingdom' AS rank FROM unit_data WHERE [kingdom] IS NOT NULL UNION ALL SELECT 'p_' + [EDIT_ATBI_RecordID], [EDIT_ATBI_RecordID], [phylum], 'phylum'‚ FROM unit_data WHERE [phylum] IS NOT NULL UNION ALL ... </pre> will create a view that meets the requirements described above. Remember that even though it looks like a table, it’s just another representation of the data stored in the table unit_data: [[File:prepEditView.png]] Such pivoting must be done in all cases when several columns need to be mapped to the same (repeatable) ABCD element. If it’s just one value to be mapped, there is no need for that. Commonly used ABCD elements that are repeatable and might ask for such preparatory transformations are * Identification * HigherTaxon * GatheringSite/NamedArea * Metadata/Scope/GeoecologicalTerms * Metadata/Scope/TaxonomicTerms * MultimediaObjects * MeasurementsOrFacts If you have problems preparing your database for publication, please [[Special:Contact|contact the BioCASe team]]. ea9147b8ecbb0729c01bb383cc1ef3320a69b384 485 484 2012-04-05T09:28:16Z JoergHoletschek 3 /* Controlled Denormalisation */ wikitext text/x-wiki Before you start publishing your database with BioCASe, you need to consider several issues. In some cases, some preparation of your database for the publication might be necessary. You should read this page of the wiki careful before you start configuring the Provider Software for a given database. == Live-DB versus Snapshot == The easiest way is to connect BioCASe to the same database you’re working with. This would result in any changes you do being visible on the BioCASe access point right away – which is an idea you might not like. Some curators prefer to publish defined versions of their database, for example after they’ve finished reviewing all specimens of a certain genus. If you prefer this, you need to create a second database that holds defined snapshots of your working database and that will be published instead. Snapshots could be generated in a regular interval (e.g., each Friday evening) or upon request. Please contact your IT guys for more help on this. == Access Control == The BioCASe Provider Software will only publish your data to biodiversity networks, meaning it will only read from the database and never do any changes. For security reasons it’s a good idea to create a user for BioCASe that has only read privileges on the database – more precisely on the tables you want to publish. Since the configuration tool will try to load a list of tables and columns from the database to allow easier setup, this user should also be allowed to load meta-information (<small>SHOW TABLES, SHOW COLUMNS</small>). == Metadata == Metadata describe your dataset as a whole - title, description, curator’s name, ownership, date of last modification, terms of use etc. BioCASe can publish a lot of metadata – and you should take the time to provide as much information as you can, since this will increase the usability of your published data. For storing metadata, please create an extra table in your database and fill it in. If you go to the BioCASe folder of your installation, you’ll find the template for this table in ''sampledb/abcdmetadata''. Either use the SQL script to create it in your database, or import it from the Access database (''metadata.mdb''). Per default, all records of your database will make up one dataset, being described by one set of metadata. In case your database holds records of several datasets, you can also group these into several datasets in ABCD by providing different sets of metadata. So when you realize during gathering of meta-information that they’re not the same for all records, this is an indicator that your database must be grouped into several datasets. If all records to be published share the same metadata, the metadata table needs to have just one row. If they don’t, it needs to hold several rows, one set of metadata for each dataset, identified by a unique primary key (1, 2, …). In order to associate each record of your database with the correct set of metadata, you will need to add a foreign key into the base record table that links to the respective entry in the metadata table. Ask your IT guys if you need help on that. == Controlled Denormalisation == As you will see during configuration, the Provider Software is able to join together the different tables of your database’s data model in order to retrieve all information to be published from the database. This works good if the number of tables is within reasonable limits, that is up to approximately 10-15. Data models used by off-the-shelf collection management system are usually highly normalised and use far more tables. Specify’s data model, for example, has about 70 tables. Configuring such a complex model in BioCASe will be a terrible pain; moreover, in case your database holds a huge number of records, the time required by the Provider Software for answering search request will become too long. Therefore it is advisable in such cases to add some controlled denormalisation to your database, that is to export the data to be published into several cache tables. These tables could be stored in the same database as the source tables or separate in an extra cache database, maybe even on a different server. They could be created in a regular interval (e.g. each night) or upon request by SQL scripts. The figure below shows this denormalisation process for the data model of Specify. On the left side, you see some of Specify’s source tables (only the ones starting with the letters A, B, C and D). With an SQL script they will be reduced to 7 cache tables, one of them being abcd_object. These tables hold only the columns that will be published, drawn from up to 15 source tables. [[image:prepDenormalisation.png|750px|border]] == Repeatable Elements in ABCD == One of the main features of ABCD (the schema you will probably use to publish your data) is repeatable elements. For example, ABCD can store multiple identifications or several URLs for multimedia objects attached to a specimen record. All these URLs will be stored in the same ABCD element – namely ''MultimediaObject/FileURI''. You can tell repeatable elements in ABCD by the existence of a container element. For ''MultimediaObject'', for example, there is a container element ''MultimediaObjects'' one level above which holds the (potentially) several multimedia objects: [[File:prepMultimediaObjects.png]] Another example is the higher taxonomy (all taxa above genus level). Typically these are stored in different columns in databases (see figure below). In ABCD, they will all end up in the same element, ''HigherTaxon/Name''. In order to be able to map the different ranks stored in columns to the same ABCD concept, we need to pivot these columns into rows – each rank kept in a column needs to be turned into a row, with two columns storing taxon name and rank, one column for linking to the according specimen record, plus one newly created primary key: [[File:prepPivoting.png]] Such a transformation can be easily done with a database view. Views do exactly what the name suggests – they provide a different view on the data. They can be used like regular tables, but they merely represent a different representation of another table. Updates of the underlying tables (or insert/deletions) will also affect views created on these tables, since the data is stored only once in the database. Let’s assume you have the higher taxonomy stored in the specimen table like this: [[File:prepEditTable.png]] The SQL statement <pre> CREATE VIEW [dbo].[vwHigherTaxa] AS SELECT 'k_' + [EDIT_ATBI_RecordID] AS id, [EDIT_ATBI_RecordID] AS unit_id, [kingdom] AS name, 'kingdom' AS rank FROM unit_data WHERE [kingdom] IS NOT NULL UNION ALL SELECT 'p_' + [EDIT_ATBI_RecordID], [EDIT_ATBI_RecordID], [phylum], 'phylum'‚ FROM unit_data WHERE [phylum] IS NOT NULL UNION ALL ... </pre> will create a view that meets the requirements described above. Remember that even though it looks like a table, it’s just another representation of the data stored in the table unit_data: [[File:prepEditView.png]] Such pivoting must be done in all cases when several columns need to be mapped to the same (repeatable) ABCD element. If it’s just one value to be mapped, there is no need for that. Commonly used ABCD elements that are repeatable and might ask for such preparatory transformations are * Identification * HigherTaxon * GatheringSite/NamedArea * Metadata/Scope/GeoecologicalTerms * Metadata/Scope/TaxonomicTerms * MultimediaObjects * MeasurementsOrFacts If you have problems preparing your database for publication, please [[Special:Contact|contact the BioCASe team]]. 77a358a48ce5602fc560292237b39e822b8bef7c 486 485 2012-04-05T09:30:36Z JoergHoletschek 3 /* Repeatable Elements in ABCD */ wikitext text/x-wiki Before you start publishing your database with BioCASe, you need to consider several issues. In some cases, some preparation of your database for the publication might be necessary. You should read this page of the wiki careful before you start configuring the Provider Software for a given database. == Live-DB versus Snapshot == The easiest way is to connect BioCASe to the same database you’re working with. This would result in any changes you do being visible on the BioCASe access point right away – which is an idea you might not like. Some curators prefer to publish defined versions of their database, for example after they’ve finished reviewing all specimens of a certain genus. If you prefer this, you need to create a second database that holds defined snapshots of your working database and that will be published instead. Snapshots could be generated in a regular interval (e.g., each Friday evening) or upon request. Please contact your IT guys for more help on this. == Access Control == The BioCASe Provider Software will only publish your data to biodiversity networks, meaning it will only read from the database and never do any changes. For security reasons it’s a good idea to create a user for BioCASe that has only read privileges on the database – more precisely on the tables you want to publish. Since the configuration tool will try to load a list of tables and columns from the database to allow easier setup, this user should also be allowed to load meta-information (<small>SHOW TABLES, SHOW COLUMNS</small>). == Metadata == Metadata describe your dataset as a whole - title, description, curator’s name, ownership, date of last modification, terms of use etc. BioCASe can publish a lot of metadata – and you should take the time to provide as much information as you can, since this will increase the usability of your published data. For storing metadata, please create an extra table in your database and fill it in. If you go to the BioCASe folder of your installation, you’ll find the template for this table in ''sampledb/abcdmetadata''. Either use the SQL script to create it in your database, or import it from the Access database (''metadata.mdb''). Per default, all records of your database will make up one dataset, being described by one set of metadata. In case your database holds records of several datasets, you can also group these into several datasets in ABCD by providing different sets of metadata. So when you realize during gathering of meta-information that they’re not the same for all records, this is an indicator that your database must be grouped into several datasets. If all records to be published share the same metadata, the metadata table needs to have just one row. If they don’t, it needs to hold several rows, one set of metadata for each dataset, identified by a unique primary key (1, 2, …). In order to associate each record of your database with the correct set of metadata, you will need to add a foreign key into the base record table that links to the respective entry in the metadata table. Ask your IT guys if you need help on that. == Controlled Denormalisation == As you will see during configuration, the Provider Software is able to join together the different tables of your database’s data model in order to retrieve all information to be published from the database. This works good if the number of tables is within reasonable limits, that is up to approximately 10-15. Data models used by off-the-shelf collection management system are usually highly normalised and use far more tables. Specify’s data model, for example, has about 70 tables. Configuring such a complex model in BioCASe will be a terrible pain; moreover, in case your database holds a huge number of records, the time required by the Provider Software for answering search request will become too long. Therefore it is advisable in such cases to add some controlled denormalisation to your database, that is to export the data to be published into several cache tables. These tables could be stored in the same database as the source tables or separate in an extra cache database, maybe even on a different server. They could be created in a regular interval (e.g. each night) or upon request by SQL scripts. The figure below shows this denormalisation process for the data model of Specify. On the left side, you see some of Specify’s source tables (only the ones starting with the letters A, B, C and D). With an SQL script they will be reduced to 7 cache tables, one of them being abcd_object. These tables hold only the columns that will be published, drawn from up to 15 source tables. [[image:prepDenormalisation.png|750px|border]] == Repeatable Elements in ABCD == One of the main features of ABCD (the schema you will probably use to publish your data) is repeatable elements. For example, ABCD can store multiple identifications or several URLs for multimedia objects attached to a specimen record. All these URLs will be stored in the same ABCD element – namely ''MultimediaObject/FileURI''. You can tell repeatable elements in ABCD by the existence of a container element. For ''MultimediaObject'', for example, there is a container element ''MultimediaObjects'' one level above which will hold the (potentially) several multimedia objects: [[File:prepMultimediaObjects.png]] Another example is the higher taxonomy (all taxa above genus level). Typically these are stored in different columns in databases (see figure below). In ABCD, they will all end up in the same element, ''HigherTaxon/Name''. In order to be able to map the different ranks stored in columns to the same ABCD concept, we need to pivot these columns into rows – each rank kept in a column needs to be turned into a row, with two columns storing taxon name and rank, one column for linking to the according specimen record, plus one newly created primary key: [[File:prepPivoting.png|border]] Such a transformation can be easily done with a database view. Views do exactly what the name suggests – they provide a different view on the data. They can be used like regular tables, but they merely represent a different representation of another table. Updates of the underlying tables (or insert/deletions) will also affect views created on these tables, since the data is stored only once in the database. Let’s assume you have the higher taxonomy stored in the specimen table like this: [[File:prepEditTable.png]] The SQL statement <pre> CREATE VIEW [dbo].[vwHigherTaxa] AS SELECT 'k_' + [EDIT_ATBI_RecordID] AS id, [EDIT_ATBI_RecordID] AS unit_id, [kingdom] AS name, 'kingdom' AS rank FROM unit_data WHERE [kingdom] IS NOT NULL UNION ALL SELECT 'p_' + [EDIT_ATBI_RecordID], [EDIT_ATBI_RecordID], [phylum], 'phylum'‚ FROM unit_data WHERE [phylum] IS NOT NULL UNION ALL ... </pre> will create a view that meets the requirements described above. Remember that even though it looks like a table, it’s just another representation of the data stored in the table unit_data: [[File:prepEditView.png]] Such pivoting must be done in all cases when several columns need to be mapped to the same (repeatable) ABCD element. If it’s just one value to be mapped, there is no need for that. Commonly used ABCD elements that are repeatable and might ask for such preparatory transformations are * Identification * HigherTaxon * GatheringSite/NamedArea * Metadata/Scope/GeoecologicalTerms * Metadata/Scope/TaxonomicTerms * MultimediaObjects * MeasurementsOrFacts If you have problems preparing your database for publication, please [[Special:Contact|contact the BioCASe team]]. 7ef645f5fc3c7c7741242b7c07ca55a8ba6ffd97 6 96 483 2012-04-05T09:26:21Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 0 94 488 468 2012-04-06T15:20:41Z FranckTheeten 15 wikitext text/x-wiki Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes. Si vous êtes un débutant, veuillez lire le Guide du Débutant [[BeginnersGuide|Beginner’s Guide]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit soit être à l'extérieur d'un pare-feu institutionnel ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de staisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine à la base de données à publier - soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fontionner également avec les versions 2.6 et 2.7, la version recommandée est de 2.5 Python (en fait la dernière sous-version stable de la branche 2.5: 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le bib site Web (par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous faites l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier Program Files par défaut. En effet à cet endroit le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir du Subversion du BGBM (Subversion ou SVN est un logiciel qui permet à plusieurs programmerus de travailelr en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la docuemntations des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (préinstallé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>. Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing) and use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python. You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). We know that configuring Python for IIS can a pain; however, there are several good tutorials on the web, for example * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a Postgres and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] Installation From Software Provider BioCASE Aller à: Navigation, rechercher Installation Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes. Si vous êtes un débutant, veuillez lire le Guide du Débutant en premier lieu. Contenu * 1 Prérequis * 2 Installation de Apache / IIS et Python * 3 Téléchargement du logiciel fournisseur de données o 3.1 Téléchargement du fichier d'archive o 3,2 téléchargement à partir du serveur Subversion * 4 Exécution du script d'installation * 5 Lier BioCASe et serveur Web o 5.1 Apache o Internet Information Server 5.2 * 6 Installation de paquets Python supplémentaires * 7 Modification du mot de passe par défaut * 8 Vérification de l'installation Prérequis Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: 1. Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. 2. Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit soit être en dehors de tout pare-feu institutionnel ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant (note : il n’est pas nécessaire de stasifaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement). 3. Une connexion directe peut être établie à partir de cette machine à la base de données à publier - soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système de l'aide, lisez la FAQ ou prenez contact avec l'équipe BioCASe. Installation de Apache / IIS et Python Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fontionner également avec les versions 2.6 et 2.7, la version recommandée est de 2.5 Python (en fait la dernière sous-version stable de la branche 2.5: 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le bib site Web (par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous faites l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier Program Files par défaut. En effet à cet endroit le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Quand vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. Téléchargement du logiciel fournisseur Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir de notre dépôt Subversion. La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. Téléchargement du fichier d'archive Il suffit de télécharger l'archive à partir de la section Téléchargement du site BioCASe et décompresser le contenu du dossier de destination. Téléchargement à partir du dépôt Subversion Si vous utilisez Windows, vous devez installer un client Subversion premier (Tortoise , qui est livré avec une jolie interface très commode). Linux et MacOS contiennent généralement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. Pour télécharger le BPS avec le client Subversion en ligne de commande (préinstallé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>. Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: Exécution du script d'installation Le script d'installation lancera les fichiers de configuration avec les valeurs que vous fournissez et liera les fichiers de script à votre installation de Python (vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement des répertoires d’Apache, Python ou du logiciel lui-même sur votre ordinateur). Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (ou «invite de commande" dans Windows, à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter »). Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS (par exemple « cd D://biocaseprovider ») et tapez: python setup.py Si la commande ne marche pas , vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python. Exemple (sous Windows et DOS, en supposant que vous avez installé python dans sa version 2.5 et dans le répertoire C:\Python25. Adaptez les valeurs en fonction de la situation sur votre ordinateur) C:\Python25\setup.py setup.py Assurez-vous que la version de Python donnée dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de l’intepréteur comme décrit ci-dessus): InstSetupScriptStart.png Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur Entrée. Domaine Webserver: Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, http://www.moninstitution.org. Vous pouvez conserver le localhost par défaut tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) plus tard. URL de base: Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par xemple http://moinstitution.org/monbiocase/ ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à www.myinstitution.org/biocase. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! 2ffdf1ffbc50b82145a1c6bbcaa56b140536e86b 489 488 2012-04-06T17:14:20Z FranckTheeten 15 wikitext text/x-wiki Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes. Si vous êtes un débutant, veuillez lire le Guide du Débutant [[BeginnersGuide|Beginner’s Guide]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit soit être à l'extérieur d'un pare-feu institutionnel ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de staisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine à la base de données à publier - soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fontionner également avec les versions 2.6 et 2.7, la version recommandée est de 2.5 Python (en fait la dernière sous-version stable de la branche 2.5: 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le bib site Web (par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous faites l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier Program Files par défaut. En effet à cet endroit le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir du Subversion du BGBM (Subversion ou SVN est un logiciel qui permet à plusieurs programmerus de travailelr en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la docuemntations des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (préinstallé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS (par exemple si votre provider est installé à l'emplacement ''D:\\biocase''<pre>D:</pre><pre>cd D://biocase</pre>. Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Exemple sous Windows et DOS, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python donnée dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de l’intepréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecutre depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur et permettre au modne extérieur de voir vos fichiers de configuration (qui incluent des mots de passe). Ainsi si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre server est ''d:\biocase\www' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement a lieu simplement en copiant les directives affichées à la fin du script d'installatioon dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache. Si vous installer BioCASe sur un serveur qui est déjà utlisé pour publier des sites webs, veuillez en informer l'administrateur avant de modifier ce fichier! [[image:instHttpdConf.png|480px]] Sauvee le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utlisez IIS vous devez créer d'abord un répertoire virtuel pour le répertoire physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire (Read, Write, Script Execution, Directory listing), utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier .cgi and .py avec le chemin de votre fichier exécutable Python. Vous devriez aussi aussi ajouter ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmoisn de bonenx explciations sur le web, par exemple * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a Postgres and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] 58eacde02ead6761c1460b956e73c5adaef3848b 490 489 2012-04-10T12:51:08Z FranckTheeten 15 wikitext text/x-wiki Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes. Si vous êtes un débutant, veuillez lire le Guide du Débutant [[BeginnersGuide|Beginner’s Guide]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit soit être à l'extérieur d'un pare-feu institutionnel ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de staisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine à la base de données à publier - soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fontionner également avec les versions 2.6 et 2.7, la version recommandée est de 2.5 Python (en fait la dernière sous-version stable de la branche 2.5: 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le bib site Web (par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous faites l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier Program Files par défaut. En effet à cet endroit le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir du Subversion du BGBM (Subversion ou SVN est un logiciel qui permet à plusieurs programmerus de travailelr en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la docuemntations des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (préinstallé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS (par exemple si votre provider est installé à l'emplacement ''D:\\biocase''<pre>D:</pre><pre>cd D://biocase</pre>. Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Exemple sous Windows et DOS, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python donnée dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de l’intepréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecutre depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur et permettre au modne extérieur de voir vos fichiers de configuration (qui incluent des mots de passe). Ainsi si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre server est ''d:\biocase\www' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement a lieu simplement en copiant les directives affichées à la fin du script d'installatioon dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache. Si vous installer BioCASe sur un serveur qui est déjà utlisé pour publier des sites webs, veuillez en informer l'administrateur avant de modifier ce fichier! [[image:instHttpdConf.png|480px]] Sauvee le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utlisez IIS vous devez créer d'abord un répertoire virtuel pour le répertoire physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'), utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier .cgi and .py avec le chemin de votre fichier exécutable Python. Vous devriez aussi aussi ajouter ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmons de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine name of the machine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'acceuil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du logiciel utilisé pour la base de données à publier et des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'acceuil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes), qui indique également si lacette librairie est installée sur le système et avec quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe libraries sont déjà inclsues dans le fichier d'archive que vous avez installé ou bien le répertoire Subversion d'où vous êtes partis. L librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos don"nes sur le Web) – ainsi si vous n'envisagez pas de l'utiliser il n'est pas nécessaire de l'installer. Sinon cliquez sur le lien 'Download', trouvez le fichier d'instllation qui convient à votre système d'exploitation et installez 4SuiteXML à partir du fichier téléchargé. Pour chaque type de base de données (SGBDR ou Système Gestionnaire bde base de donénes Relationnelle dans le jargon), vous devrez installer le packager ou driver Python correspondant. Si votre collection est par exemple stockée dans une base de Donénes MySQL, cliquez sur le lien pour télécharger le module MySQL, trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraries dans votre navigateur. La ligne de la liste qui correspond au drviver devrait passer de l'orange pour 'non installé' à uen indication verte contenent la version de la librarie. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utlisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessire d'installer Graphviz. Dans le cas contraire nous vous recommendons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquaut, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les pramètres du BPS. A la fin de la procédure de configuration, vous devriez changer le mot de passe. Retournez à l'écran d'acceuil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la magde Configuration dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation (nosu supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''). A la quatrième ligne, vous pouvez encoder un nouveau mot de passe. Soyez précautionneux en le tapant (si jamais vous l'oubliez, il est toujours possible de le réintialiser en modifiant directement le fichier de configuration). Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers bionaires, car Graphviz n'est pas une libraries Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sut votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary<.small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''), sous Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connection à une base de donénes), vous devriez vous assurer que tout est installé correctement: # Taper ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à voitre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modificables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les Logs (fichiers qui enregistrent les opérations sur la machine) et la configuration. Sinon, le chemin vers le fichier posent problème sera montré –. Veuillez vous assurer que Python et.ou Apache peuvent écrire sur ce fichier (par exemple sur Ubuntu où Apache correspond à un compte utlsiateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre>). Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connection via la librairie ODBC) et que vous ne souhaitez pas utliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] beffe5eecc8d8420e283d633f4ea9a5c57fbb4d7 491 490 2012-04-10T12:51:42Z FranckTheeten 15 /* Apache */ wikitext text/x-wiki Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes. Si vous êtes un débutant, veuillez lire le Guide du Débutant [[BeginnersGuide|Beginner’s Guide]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit soit être à l'extérieur d'un pare-feu institutionnel ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de staisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine à la base de données à publier - soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fontionner également avec les versions 2.6 et 2.7, la version recommandée est de 2.5 Python (en fait la dernière sous-version stable de la branche 2.5: 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le bib site Web (par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous faites l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier Program Files par défaut. En effet à cet endroit le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir du Subversion du BGBM (Subversion ou SVN est un logiciel qui permet à plusieurs programmerus de travailelr en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la docuemntations des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (préinstallé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS (par exemple si votre provider est installé à l'emplacement ''D:\\biocase''<pre>D:</pre><pre>cd D://biocase</pre>. Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Exemple sous Windows et DOS, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python donnée dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de l’intepréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecutre depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur et permettre au modne extérieur de voir vos fichiers de configuration (qui incluent des mots de passe). Ainsi si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre server est ''d:\biocase\www' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache. Si vous installer BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez en informer l'administrateur avant de modifier ce fichier! [[image:instHttpdConf.png|480px]] Sauvee le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utlisez IIS vous devez créer d'abord un répertoire virtuel pour le répertoire physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'), utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier .cgi and .py avec le chemin de votre fichier exécutable Python. Vous devriez aussi aussi ajouter ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmons de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine name of the machine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'acceuil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du logiciel utilisé pour la base de données à publier et des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'acceuil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes), qui indique également si lacette librairie est installée sur le système et avec quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe libraries sont déjà inclsues dans le fichier d'archive que vous avez installé ou bien le répertoire Subversion d'où vous êtes partis. L librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos don"nes sur le Web) – ainsi si vous n'envisagez pas de l'utiliser il n'est pas nécessaire de l'installer. Sinon cliquez sur le lien 'Download', trouvez le fichier d'instllation qui convient à votre système d'exploitation et installez 4SuiteXML à partir du fichier téléchargé. Pour chaque type de base de données (SGBDR ou Système Gestionnaire bde base de donénes Relationnelle dans le jargon), vous devrez installer le packager ou driver Python correspondant. Si votre collection est par exemple stockée dans une base de Donénes MySQL, cliquez sur le lien pour télécharger le module MySQL, trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraries dans votre navigateur. La ligne de la liste qui correspond au drviver devrait passer de l'orange pour 'non installé' à uen indication verte contenent la version de la librarie. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utlisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessire d'installer Graphviz. Dans le cas contraire nous vous recommendons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquaut, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les pramètres du BPS. A la fin de la procédure de configuration, vous devriez changer le mot de passe. Retournez à l'écran d'acceuil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la magde Configuration dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation (nosu supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''). A la quatrième ligne, vous pouvez encoder un nouveau mot de passe. Soyez précautionneux en le tapant (si jamais vous l'oubliez, il est toujours possible de le réintialiser en modifiant directement le fichier de configuration). Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers bionaires, car Graphviz n'est pas une libraries Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sut votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary<.small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''), sous Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connection à une base de donénes), vous devriez vous assurer que tout est installé correctement: # Taper ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à voitre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modificables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les Logs (fichiers qui enregistrent les opérations sur la machine) et la configuration. Sinon, le chemin vers le fichier posent problème sera montré –. Veuillez vous assurer que Python et.ou Apache peuvent écrire sur ce fichier (par exemple sur Ubuntu où Apache correspond à un compte utlsiateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre>). Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connection via la librairie ODBC) et que vous ne souhaitez pas utliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] 5fa0ffaf35ba6e9f88b89b2a9a02104153837e6b 492 491 2012-04-10T12:52:18Z FranckTheeten 15 /* Vérifiez votre Installation */ wikitext text/x-wiki Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes. Si vous êtes un débutant, veuillez lire le Guide du Débutant [[BeginnersGuide|Beginner’s Guide]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit soit être à l'extérieur d'un pare-feu institutionnel ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de staisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine à la base de données à publier - soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fontionner également avec les versions 2.6 et 2.7, la version recommandée est de 2.5 Python (en fait la dernière sous-version stable de la branche 2.5: 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le bib site Web (par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous faites l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier Program Files par défaut. En effet à cet endroit le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir du Subversion du BGBM (Subversion ou SVN est un logiciel qui permet à plusieurs programmerus de travailelr en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la docuemntations des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (préinstallé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS (par exemple si votre provider est installé à l'emplacement ''D:\\biocase''<pre>D:</pre><pre>cd D://biocase</pre>. Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Exemple sous Windows et DOS, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python donnée dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de l’intepréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecutre depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur et permettre au modne extérieur de voir vos fichiers de configuration (qui incluent des mots de passe). Ainsi si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre server est ''d:\biocase\www' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache. Si vous installer BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez en informer l'administrateur avant de modifier ce fichier! [[image:instHttpdConf.png|480px]] Sauvee le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utlisez IIS vous devez créer d'abord un répertoire virtuel pour le répertoire physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'), utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier .cgi and .py avec le chemin de votre fichier exécutable Python. Vous devriez aussi aussi ajouter ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmons de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine name of the machine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'acceuil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du logiciel utilisé pour la base de données à publier et des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'acceuil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes), qui indique également si lacette librairie est installée sur le système et avec quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe libraries sont déjà inclsues dans le fichier d'archive que vous avez installé ou bien le répertoire Subversion d'où vous êtes partis. L librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos don"nes sur le Web) – ainsi si vous n'envisagez pas de l'utiliser il n'est pas nécessaire de l'installer. Sinon cliquez sur le lien 'Download', trouvez le fichier d'instllation qui convient à votre système d'exploitation et installez 4SuiteXML à partir du fichier téléchargé. Pour chaque type de base de données (SGBDR ou Système Gestionnaire bde base de donénes Relationnelle dans le jargon), vous devrez installer le packager ou driver Python correspondant. Si votre collection est par exemple stockée dans une base de Donénes MySQL, cliquez sur le lien pour télécharger le module MySQL, trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraries dans votre navigateur. La ligne de la liste qui correspond au drviver devrait passer de l'orange pour 'non installé' à uen indication verte contenent la version de la librarie. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utlisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessire d'installer Graphviz. Dans le cas contraire nous vous recommendons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquaut, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les pramètres du BPS. A la fin de la procédure de configuration, vous devriez changer le mot de passe. Retournez à l'écran d'acceuil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la magde Configuration dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation (nosu supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''). A la quatrième ligne, vous pouvez encoder un nouveau mot de passe. Soyez précautionneux en le tapant (si jamais vous l'oubliez, il est toujours possible de le réintialiser en modifiant directement le fichier de configuration). Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers bionaires, car Graphviz n'est pas une libraries Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sut votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary<.small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''), sous Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connection à une base de donénes), vous devriez vous assurer que tout est installé correctement: # Taper ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à voitre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[##Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modificables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les Logs (fichiers qui enregistrent les opérations sur la machine) et la configuration. Sinon, le chemin vers le fichier posent problème sera montré –. Veuillez vous assurer que Python et.ou Apache peuvent écrire sur ce fichier (par exemple sur Ubuntu où Apache correspond à un compte utlsiateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre>). Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connection via la librairie ODBC) et que vous ne souhaitez pas utliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] bc9dc4ca2d66deb1a406f5189dd0cfa91573341a 493 492 2012-04-10T12:52:52Z FranckTheeten 15 /* Vérifiez votre Installation */ wikitext text/x-wiki Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes. Si vous êtes un débutant, veuillez lire le Guide du Débutant [[BeginnersGuide|Beginner’s Guide]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit soit être à l'extérieur d'un pare-feu institutionnel ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de staisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine à la base de données à publier - soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fontionner également avec les versions 2.6 et 2.7, la version recommandée est de 2.5 Python (en fait la dernière sous-version stable de la branche 2.5: 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le bib site Web (par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous faites l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier Program Files par défaut. En effet à cet endroit le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir du Subversion du BGBM (Subversion ou SVN est un logiciel qui permet à plusieurs programmerus de travailelr en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la docuemntations des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (préinstallé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS (par exemple si votre provider est installé à l'emplacement ''D:\\biocase''<pre>D:</pre><pre>cd D://biocase</pre>. Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Exemple sous Windows et DOS, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python donnée dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de l’intepréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecutre depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur et permettre au modne extérieur de voir vos fichiers de configuration (qui incluent des mots de passe). Ainsi si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre server est ''d:\biocase\www' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache. Si vous installer BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez en informer l'administrateur avant de modifier ce fichier! [[image:instHttpdConf.png|480px]] Sauvee le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utlisez IIS vous devez créer d'abord un répertoire virtuel pour le répertoire physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'), utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier .cgi and .py avec le chemin de votre fichier exécutable Python. Vous devriez aussi aussi ajouter ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmons de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine name of the machine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'acceuil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du logiciel utilisé pour la base de données à publier et des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'acceuil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes), qui indique également si lacette librairie est installée sur le système et avec quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe libraries sont déjà inclsues dans le fichier d'archive que vous avez installé ou bien le répertoire Subversion d'où vous êtes partis. L librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos don"nes sur le Web) – ainsi si vous n'envisagez pas de l'utiliser il n'est pas nécessaire de l'installer. Sinon cliquez sur le lien 'Download', trouvez le fichier d'instllation qui convient à votre système d'exploitation et installez 4SuiteXML à partir du fichier téléchargé. Pour chaque type de base de données (SGBDR ou Système Gestionnaire bde base de donénes Relationnelle dans le jargon), vous devrez installer le packager ou driver Python correspondant. Si votre collection est par exemple stockée dans une base de Donénes MySQL, cliquez sur le lien pour télécharger le module MySQL, trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraries dans votre navigateur. La ligne de la liste qui correspond au drviver devrait passer de l'orange pour 'non installé' à uen indication verte contenent la version de la librarie. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utlisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessire d'installer Graphviz. Dans le cas contraire nous vous recommendons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquaut, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les pramètres du BPS. A la fin de la procédure de configuration, vous devriez changer le mot de passe. Retournez à l'écran d'acceuil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la magde Configuration dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation (nosu supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''). A la quatrième ligne, vous pouvez encoder un nouveau mot de passe. Soyez précautionneux en le tapant (si jamais vous l'oubliez, il est toujours possible de le réintialiser en modifiant directement le fichier de configuration). Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers bionaires, car Graphviz n'est pas une libraries Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sut votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary<.small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''), sous Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connection à une base de donénes), vous devriez vous assurer que tout est installé correctement: # Taper ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à voitre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modificables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les Logs (fichiers qui enregistrent les opérations sur la machine) et la configuration. Sinon, le chemin vers le fichier posent problème sera montré –. Veuillez vous assurer que Python et.ou Apache peuvent écrire sur ce fichier (par exemple sur Ubuntu où Apache correspond à un compte utlsiateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre>). Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connection via la librairie ODBC) et que vous ne souhaitez pas utliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] 6a9ca42ffc94164ee709d9a0d8fd8a8e1650a90a 494 493 2012-04-10T12:58:52Z FranckTheeten 15 wikitext text/x-wiki Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes. Si vous êtes un débutant, veuillez lire le Guide du Débutant [[BeginnersGuide|Beginner’s Guide]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit soit être à l'extérieur d'un pare-feu institutionnel ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de satisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine à la base de données à publier - soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fonctionner également avec les versions 2.6 et 2.7, la version recommandée est de 2.5 Python (en fait la dernière sous-version stable de la branche 2.5: 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le bib site Web (par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous faites l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier Program Files par défaut. En effet à cet endroit le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir du Subversion du BGBM (Subversion ou SVN est un logiciel qui permet à plusieurs programmeurs de travailler en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la documentation des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (pré-installé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS (par exemple si votre provider est installé à l'emplacement ''D:\\biocase''<pre>D:</pre><pre>cd D://biocase</pre>. Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Exemple sous Windows et DOS, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python donnée dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de interpréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecture depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur et permettre au modne extérieur de voir vos fichiers de configuration (qui incluent des mots de passe). Ainsi si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre server est ''d:\biocase\www' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache. Si vous installer BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez en informer l'administrateur avant de modifier ce fichier! [[image:instHttpdConf.png|480px]] Sauvez le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utlisez IIS vous devez créer d'abord un répertoire virtuel pour le répertoire physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'), utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier .cgi and .py avec le chemin de votre fichier exécutable Python. Vous devriez aussi aussi ajouter ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmons de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'accueil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du logiciel utilisé pour la base de données à publier et des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'accueil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes), qui indique également si cette librairie est présente sur le système et, si ou, dans quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci: [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe proprement dite sont déjà inclues dans le fichier d'archive que vous avez installé ou bien le répertoire Subversion d'où vous êtes partis. La librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos don"nes sur le Web) – ainsi si vous n'envisagez pas de l'utiliser il n'est pas nécessaire de l'installer. Sinon cliquez sur le lien 'Download', trouvez le fichier d'installation qui convient à votre système d'exploitation et installez 4SuiteXML à partir du fichier téléchargé. Néanmoins, Pour chaque type de base de données (SGBDR ou Système Gestionnaire be base de données Relationnelle dans le jargon), vous devrez installer le package ou driver Python correspondant. Ce driver permet à un programme Python de se connecter à une base de donénes . Il doit être installé et configuré non pas au niveau de BioCASe, mais à celui de l'environnement Python; mais néanmoins le BPS peut vous guider dans leur installation. Si votre collection est par exemple stockée dans une base de données MySQL, cliquez sur le lien pour télécharger le module MySQL, trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraires dans votre navigateur. La ligne de la liste qui correspond au driver devrait passer de l'orange pour 'non installé' à une indication verte mentionnant la version de la libraire. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utilisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessire d'installer Graphviz. Dans le cas contraire nous vous recommandons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquat, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les pramètres du BPS. A la fin de la procédure de configuration, vous devriez changer le mot de passe. Retournez à l'écran d'accueil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la magde Configuration dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation (nosu supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''). A la quatrième ligne, vous pouvez encoder un nouveau mot de passe. Soyez précautionneux en le tapant (si jamais vous l'oubliez, il est toujours possible de le réintialiser en modifiant directement le fichier de configuration). Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers binaires, car Graphviz n'est pas une librairie Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sut votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary<.small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''), sous Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connection à une base de donénes), vous devriez vous assurer que tout est installé correctement: # Taper ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à votre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modificables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les Logs (fichiers qui enregistrent les opérations sur la machine) et la configuration. Sinon, le chemin vers le fichier posent problème sera montré –. Veuillez vous assurer que Python et.ou Apache peuvent écrire sur ce fichier (par exemple sur Ubuntu où Apache correspond à un compte utlsiateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre>). Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connexion via la librairie ODBC) et que vous ne souhaitez pas utiliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] fa0ca5ba40076cc77473359efff7da8a7b598d48 495 494 2012-04-10T12:59:16Z FranckTheeten 15 /* Changer le mot de passe par défaut */ wikitext text/x-wiki Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes. Si vous êtes un débutant, veuillez lire le Guide du Débutant [[BeginnersGuide|Beginner’s Guide]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit soit être à l'extérieur d'un pare-feu institutionnel ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de satisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine à la base de données à publier - soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fonctionner également avec les versions 2.6 et 2.7, la version recommandée est de 2.5 Python (en fait la dernière sous-version stable de la branche 2.5: 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le bib site Web (par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous faites l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier Program Files par défaut. En effet à cet endroit le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir du Subversion du BGBM (Subversion ou SVN est un logiciel qui permet à plusieurs programmeurs de travailler en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la documentation des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (pré-installé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS (par exemple si votre provider est installé à l'emplacement ''D:\\biocase''<pre>D:</pre><pre>cd D://biocase</pre>. Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Exemple sous Windows et DOS, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python donnée dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de interpréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecture depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur et permettre au modne extérieur de voir vos fichiers de configuration (qui incluent des mots de passe). Ainsi si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre server est ''d:\biocase\www' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache. Si vous installer BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez en informer l'administrateur avant de modifier ce fichier! [[image:instHttpdConf.png|480px]] Sauvez le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utlisez IIS vous devez créer d'abord un répertoire virtuel pour le répertoire physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'), utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier .cgi and .py avec le chemin de votre fichier exécutable Python. Vous devriez aussi aussi ajouter ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmons de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'accueil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du logiciel utilisé pour la base de données à publier et des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'accueil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes), qui indique également si cette librairie est présente sur le système et, si ou, dans quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci: [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe proprement dite sont déjà inclues dans le fichier d'archive que vous avez installé ou bien le répertoire Subversion d'où vous êtes partis. La librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos don"nes sur le Web) – ainsi si vous n'envisagez pas de l'utiliser il n'est pas nécessaire de l'installer. Sinon cliquez sur le lien 'Download', trouvez le fichier d'installation qui convient à votre système d'exploitation et installez 4SuiteXML à partir du fichier téléchargé. Néanmoins, Pour chaque type de base de données (SGBDR ou Système Gestionnaire be base de données Relationnelle dans le jargon), vous devrez installer le package ou driver Python correspondant. Ce driver permet à un programme Python de se connecter à une base de donénes . Il doit être installé et configuré non pas au niveau de BioCASe, mais à celui de l'environnement Python; mais néanmoins le BPS peut vous guider dans leur installation. Si votre collection est par exemple stockée dans une base de données MySQL, cliquez sur le lien pour télécharger le module MySQL, trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraires dans votre navigateur. La ligne de la liste qui correspond au driver devrait passer de l'orange pour 'non installé' à une indication verte mentionnant la version de la libraire. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utilisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessire d'installer Graphviz. Dans le cas contraire nous vous recommandons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquat, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les pramètres du BPS. A la fin de la procédure de configuration, vous devriez changer le mot de passe. Retournez à l'écran d'accueil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la magde Configuration dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation (nosu supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''). A la quatrième ligne, vous pouvez encoder un nouveau mot de passe. Soyez précautionneux en le tapant (si jamais vous l'oubliez, il est toujours possible de le réintialiser en modifiant directement le fichier de configuration). Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers binaires, car Graphviz n'est pas une librairie Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sut votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary</small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''), sous Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connection à une base de donénes), vous devriez vous assurer que tout est installé correctement: # Taper ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à votre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modificables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les Logs (fichiers qui enregistrent les opérations sur la machine) et la configuration. Sinon, le chemin vers le fichier posent problème sera montré –. Veuillez vous assurer que Python et.ou Apache peuvent écrire sur ce fichier (par exemple sur Ubuntu où Apache correspond à un compte utlsiateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre>). Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connexion via la librairie ODBC) et que vous ne souhaitez pas utiliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] 4a5d0e0d1434f8cce663d0279235ca07be299105 496 495 2012-04-10T13:01:02Z FranckTheeten 15 wikitext text/x-wiki '''Traduction: Franck Theeten Musée Royal de l'Afrique Centrale de Tervuren Pour le projet CABIN (Central African Biodiversity Information Network)''' Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes en fonction de votre degré de familiarité avec Python et Apache. Si vous êtes un débutant, veuillez lire le Guide du Débutant [[BeginnersGuide|Beginner’s Guide]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit soit être à l'extérieur d'un pare-feu institutionnel ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de satisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine à la base de données à publier - soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fonctionner également avec les versions 2.6 et 2.7, la version recommandée est de 2.5 Python (en fait la dernière sous-version stable de la branche 2.5: 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le bib site Web (par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous faites l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier Program Files par défaut. En effet à cet endroit le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir du Subversion du BGBM (Subversion ou SVN est un logiciel qui permet à plusieurs programmeurs de travailler en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la documentation des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (pré-installé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS (par exemple si votre provider est installé à l'emplacement ''D:\\biocase''<pre>D:</pre><pre>cd D://biocase</pre>. Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Exemple sous Windows et DOS, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python donnée dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de interpréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecture depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur et permettre au modne extérieur de voir vos fichiers de configuration (qui incluent des mots de passe). Ainsi si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre server est ''d:\biocase\www' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache. Si vous installer BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez en informer l'administrateur avant de modifier ce fichier! [[image:instHttpdConf.png|480px]] Sauvez le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utlisez IIS vous devez créer d'abord un répertoire virtuel pour le répertoire physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'), utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier .cgi and .py avec le chemin de votre fichier exécutable Python. Vous devriez aussi aussi ajouter ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmons de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'accueil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du logiciel utilisé pour la base de données à publier et des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'accueil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes), qui indique également si cette librairie est présente sur le système et, si ou, dans quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci: [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe proprement dite sont déjà inclues dans le fichier d'archive que vous avez installé ou bien le répertoire Subversion d'où vous êtes partis. La librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos don"nes sur le Web) – ainsi si vous n'envisagez pas de l'utiliser il n'est pas nécessaire de l'installer. Sinon cliquez sur le lien 'Download', trouvez le fichier d'installation qui convient à votre système d'exploitation et installez 4SuiteXML à partir du fichier téléchargé. Néanmoins, Pour chaque type de base de données (SGBDR ou Système Gestionnaire be base de données Relationnelle dans le jargon), vous devrez installer le package ou driver Python correspondant. Ce driver permet à un programme Python de se connecter à une base de donénes . Il doit être installé et configuré non pas au niveau de BioCASe, mais à celui de l'environnement Python; mais néanmoins le BPS peut vous guider dans leur installation. Si votre collection est par exemple stockée dans une base de données MySQL, cliquez sur le lien pour télécharger le module MySQL, trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraires dans votre navigateur. La ligne de la liste qui correspond au driver devrait passer de l'orange pour 'non installé' à une indication verte mentionnant la version de la libraire. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utilisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessire d'installer Graphviz. Dans le cas contraire nous vous recommandons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquat, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les pramètres du BPS. A la fin de la procédure de configuration, vous devriez changer le mot de passe. Retournez à l'écran d'accueil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la magde Configuration dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation (nosu supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''). A la quatrième ligne, vous pouvez encoder un nouveau mot de passe. Soyez précautionneux en le tapant (si jamais vous l'oubliez, il est toujours possible de le réintialiser en modifiant directement le fichier de configuration). Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers binaires, car Graphviz n'est pas une librairie Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sut votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary</small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''), sous Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connection à une base de donénes), vous devriez vous assurer que tout est installé correctement: # Taper ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à votre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modificables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les Logs (fichiers qui enregistrent les opérations sur la machine) et la configuration. Sinon, le chemin vers le fichier posent problème sera montré –. Veuillez vous assurer que Python et.ou Apache peuvent écrire sur ce fichier (par exemple sur Ubuntu où Apache correspond à un compte utlsiateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre>). Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connexion via la librairie ODBC) et que vous ne souhaitez pas utiliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] 9aff168e90d93465700867c27e2c46005f19dc7c 497 496 2012-04-10T13:01:38Z FranckTheeten 15 wikitext text/x-wiki '''Traduction: Franck Theeten Musée Royal de l'Afrique Centrale de Tervuren Pour le projet CABIN (Central African Biodiversity Information Network)''' Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes en fonction de votre degré de familiarité avec Python et Apache. Si vous êtes un débutant, veuillez lire le Guide du Débutant [[BeginnersGuide|Beginner’s Guide]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit soit être à l'extérieur d'un pare-feu institutionnel ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de satisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine à la base de données à publier - soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fonctionner également avec les versions 2.6 et 2.7, la version recommandée est de 2.5 Python (en fait la dernière sous-version stable de la branche 2.5: 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le bib site Web (par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous faites l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier Program Files par défaut. En effet à cet endroit le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir du Subversion du BGBM (Subversion ou SVN est un logiciel qui permet à plusieurs programmeurs de travailler en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la documentation des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (pré-installé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS (par exemple si votre provider est installé à l'emplacement ''D:\\biocase''<pre>D:</pre><pre>cd D://biocase</pre>. Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Exemple sous Windows et DOS, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python donnée dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de interpréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecture depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur et permettre au modne extérieur de voir vos fichiers de configuration (qui incluent des mots de passe). Ainsi si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre server est ''d:\biocase\www' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache. Si vous installer BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez en informer l'administrateur avant de modifier ce fichier! [[image:instHttpdConf.png|480px]] Sauvez le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utlisez IIS vous devez créer d'abord un répertoire virtuel pour le répertoire physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'), utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier .cgi and .py avec le chemin de votre fichier exécutable Python. Vous devriez aussi aussi ajouter ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmons de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'accueil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du logiciel utilisé pour la base de données à publier et des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'accueil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes), qui indique également si cette librairie est présente sur le système et, si ou, dans quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci: [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe proprement dite sont déjà inclues dans le fichier d'archive que vous avez installé ou bien le répertoire Subversion d'où vous êtes partis. La librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos don"nes sur le Web) – ainsi si vous n'envisagez pas de l'utiliser il n'est pas nécessaire de l'installer. Sinon cliquez sur le lien 'Download', trouvez le fichier d'installation qui convient à votre système d'exploitation et installez 4SuiteXML à partir du fichier téléchargé. Néanmoins, Pour chaque type de base de données (SGBDR ou Système Gestionnaire be base de données Relationnelle dans le jargon), vous devrez installer le package ou driver Python correspondant. Ce driver permet à un programme Python de se connecter à une base de donénes . Il doit être installé et configuré non pas au niveau de BioCASe, mais à celui de l'environnement Python; mais néanmoins le BPS peut vous guider dans leur installation. Si votre collection est par exemple stockée dans une base de données MySQL, cliquez sur le lien pour télécharger le module MySQL, trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraires dans votre navigateur. La ligne de la liste qui correspond au driver devrait passer de l'orange pour 'non installé' à une indication verte mentionnant la version de la libraire. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utilisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessire d'installer Graphviz. Dans le cas contraire nous vous recommandons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquat, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les pramètres du BPS. A la fin de la procédure de configuration, vous devriez changer le mot de passe. Retournez à l'écran d'accueil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la magde Configuration dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation (nosu supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''). A la quatrième ligne, vous pouvez encoder un nouveau mot de passe. Soyez précautionneux en le tapant (si jamais vous l'oubliez, il est toujours possible de le réintialiser en modifiant directement le fichier de configuration). Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers binaires, car Graphviz n'est pas une librairie Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sut votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary</small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''), sous Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connection à une base de donénes), vous devriez vous assurer que tout est installé correctement: # Taper ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à votre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modificables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les Logs (fichiers qui enregistrent les opérations sur la machine) et la configuration. Sinon, le chemin vers le fichier posent problème sera montré –. Veuillez vous assurer que Python et.ou Apache peuvent écrire sur ce fichier (par exemple sur Ubuntu où Apache correspond à un compte utlsiateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre>). Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connexion via la librairie ODBC) et que vous ne souhaitez pas utiliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] 91cd79887b258b48f9af9c19df6a28369b360368 498 497 2012-04-10T13:02:30Z FranckTheeten 15 wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes en fonction de votre degré de familiarité avec Python et Apache. Si vous êtes un débutant, veuillez lire le Guide du Débutant [[BeginnersGuide|Beginner’s Guide]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit soit être à l'extérieur d'un pare-feu institutionnel ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de satisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine à la base de données à publier - soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL Server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fonctionner également avec les versions 2.6 et 2.7, la version recommandée est de 2.5 Python (en fait la dernière sous-version stable de la branche 2.5: 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le bib site Web (par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous faites l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier Program Files par défaut. En effet à cet endroit le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir du Subversion du BGBM (Subversion ou SVN est un logiciel qui permet à plusieurs programmeurs de travailler en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la documentation des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (pré-installé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS (par exemple si votre provider est installé à l'emplacement ''D:\\biocase''<pre>D:</pre><pre>cd D://biocase</pre>. Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Exemple sous Windows et DOS, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python donnée dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de interpréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecture depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur et permettre au modne extérieur de voir vos fichiers de configuration (qui incluent des mots de passe). Ainsi si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre server est ''d:\biocase\www' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache. Si vous installer BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez en informer l'administrateur avant de modifier ce fichier! [[image:instHttpdConf.png|480px]] Sauvez le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utlisez IIS vous devez créer d'abord un répertoire virtuel pour le répertoire physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'), utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier .cgi and .py avec le chemin de votre fichier exécutable Python. Vous devriez aussi aussi ajouter ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmons de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'accueil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du logiciel utilisé pour la base de données à publier et des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'accueil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes), qui indique également si cette librairie est présente sur le système et, si ou, dans quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci: [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe proprement dite sont déjà inclues dans le fichier d'archive que vous avez installé ou bien le répertoire Subversion d'où vous êtes partis. La librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos don"nes sur le Web) – ainsi si vous n'envisagez pas de l'utiliser il n'est pas nécessaire de l'installer. Sinon cliquez sur le lien 'Download', trouvez le fichier d'installation qui convient à votre système d'exploitation et installez 4SuiteXML à partir du fichier téléchargé. Néanmoins, Pour chaque type de base de données (SGBDR ou Système Gestionnaire be base de données Relationnelle dans le jargon), vous devrez installer le package ou driver Python correspondant. Ce driver permet à un programme Python de se connecter à une base de donénes . Il doit être installé et configuré non pas au niveau de BioCASe, mais à celui de l'environnement Python; mais néanmoins le BPS peut vous guider dans leur installation. Si votre collection est par exemple stockée dans une base de données MySQL, cliquez sur le lien pour télécharger le module MySQL, trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraires dans votre navigateur. La ligne de la liste qui correspond au driver devrait passer de l'orange pour 'non installé' à une indication verte mentionnant la version de la libraire. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utilisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessire d'installer Graphviz. Dans le cas contraire nous vous recommandons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquat, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les pramètres du BPS. A la fin de la procédure de configuration, vous devriez changer le mot de passe. Retournez à l'écran d'accueil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la magde Configuration dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation (nosu supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''). A la quatrième ligne, vous pouvez encoder un nouveau mot de passe. Soyez précautionneux en le tapant (si jamais vous l'oubliez, il est toujours possible de le réintialiser en modifiant directement le fichier de configuration). Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers binaires, car Graphviz n'est pas une librairie Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sut votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary</small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''), sous Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connection à une base de donénes), vous devriez vous assurer que tout est installé correctement: # Taper ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à votre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modificables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les Logs (fichiers qui enregistrent les opérations sur la machine) et la configuration. Sinon, le chemin vers le fichier posent problème sera montré –. Veuillez vous assurer que Python et.ou Apache peuvent écrire sur ce fichier (par exemple sur Ubuntu où Apache correspond à un compte utlsiateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre>). Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connexion via la librairie ODBC) et que vous ne souhaitez pas utiliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] 23ef0a3744c77a90466d9863c72f1e014d608a2e 0 93 500 465 2012-04-10T15:16:35Z FranckTheeten 15 wikitext text/x-wiki == Qu'est ce que le Logiciel Fournisseur de données BioCASe? == Le logiciel fournisseur de données BioCASe (abrégé '''BPS'' pour '''BioCASe Provider Software''' en anglais ) est un middleware (c'est à direlogiciel placé à une position intermédiaire entre une base de données et une interface graphique) produisant des données en XML, afin de le publier sur un réseau d'information scientifique. Après avoir installé BioCASe et l'avoir configuré pour un type de base de données particulier, l'information publiée sera accessible sous la forme d'un web service de type BioCASe. Cela veut dire que l'information pourra être interrogée et recherchée avec des requêtes au format BioCASe, car BioCASe est avant tout un protocole XML pour envoyer des messages à un web service. L'intérêt de ce type de logiciel est qu'il est "agnostique" par rapport au modèle de données de la base de données d'origine, c'est à dire que les requêtes envoyées au web servcies n'ont pas besoind e connaître la strucutre de la base de données pour fonctionner. Il peut être aussi utilisé avec n'importe quel type de schéma conceptuel de données au niveau XML. Le BPS est un ensemble d'outils qui doivent être installée sur un serveur Web installé chez le fournisseur de données. Son composant central est le '''PyWrapper''', une interface XML/CGI pour se connecter à des bases de données écrite en Python. Il petmet un accès standardisé (c'est à dire via des procédures uniformes qui peuevnt être ainsi partagée par tous els acturs d'unr éseau d'information) à une large gamme de systèmes de base de données dont la structure est arbitraire (c'est à dire définie par le fournisseur de données seul). Un certain nombre d'outils est réuni autour du BPS pour configurer, tester et débugger une installation ou un web service BioCASe. Même si BioCASe peut être utilisé avec n'importe quel schéma conceptuel défini en XML, son domaine de prédilection est la publication de données d'occurrence provenant de bases de données content des spécimen ou des observations sur des réseaux d'information primaire sur la biodiversité, comme le [http://www.biocase.org réseau BioCASe] et le [http://www.gbif.org GBIF (Global Biodiversity Information Facility)]. == Prérequis pour utiliser le BPS == Avant d'utliser le logiciel fournisseur de données BioCASe, trois conditions doivent être vérifiées: * D'une part, les données à publier doivent être stockées dans une système gestionnaires de base de données relationnelle (parfois abrégé '''SGBDR''' ou '''RDBMS''' en anglais) compatible avec le langage '''SQL''', et dont le modèle de données est bien documenté, ou, au minimum, présent à l'esprit d'un gestionnaire de données. Les bases de donénes non compatibles avec SQL ne sont pas supportées par BioCASe. Actuellement, le logiciel peut se connecter au bases de données suivantes: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * Pour installer BioCASe, vous devez avoir un logiciel de serveur Web (Apache ou Microsoft IIS) accessible en permenante depuis l'Internet. Cela peut être le serveur d'une institution académique ou scientifique, ou un service d'hébérgement (même s'il est parfois pénible dans le second cas d'obtenir les droits d'installation auprès d'un service d'hébergement. Une connection commutée (peer to peer ou 'Dial-up') ne fonctionnera pas. Notez qu'il est tout à fait possuble à des fins d'entraînement, de formation ou de développment, d'installer un serveur Internet sur votre propre machine, isolé du reste de l'Internet, et de faire fonctionner BiocASe dessus. Il sera alors accessible localement pour l'ordinateur seum (via le nom http://localhost ou l'adresse IP réservée 127.0.0.1) * Vous devez avoir la possibilité d'installer Python sur le serveur web en question (si jamais ce n'est pas le cas) ainsi que des libraires ou paquets complémentaires Python. Cela peut être fait par un informaticien ou quelqu'un ayant de bonnes connaissances techniques, mais assurez vous que cela ne pose pas problèmes par rapport aux procédures et règles techniques qui pourraient être en vigeur dans votre institution. == Etapes == Une fois que vous vous êtes assurez que le BioCASe Provider Software est un choix pertinent pour vos besoins ([[Special:Contact|contactez l'équipe BioCASe]] si vous avez encore besoin d'information avant de prendre une décicion), vous devez exécutez les étapes suivantes: # Pensez à l'information que vous souhaitez publier sur le réseau. Cela implique d'avoir une réponse à la question de la propriété des données et des termes d'utlsiation, licences, copyright et restrictions dans l'utlisation qui valent pour les données publiées. Vous devriez aussi vous demander si certaines information ne devraient '''pas''' être publiées car cela pourrait avoir une conséquence sur des espèces menacées (une solution possible est de soustraire certaines occurrence du jeu de données publier ou de rendre moins précise une information exacte sur la localité d'observation). # Préparez la base de données comme décrit dand le tutoriel [[Preparation]]. Il faut alors décider de publier soit directement la base de données principale qui est utlisée pour les mises à jour, soit un 'snapshot' (une copie de tout ou une partie des données prise à un moment précis), de créer uen table de métadonnées (informations sur la base de donénes elle-même) et, dans certains cas, de transformer la présentation des données. # Suivez le tutoriel d'[[Installation]] pour installer le Fournisseur de données BioCASe. # Configurez un jeu de données BioCASe dans le logiciel et connectez-le à votre base de données. Configuez alors le jeu de données comme décrit dans le tutoriel [[DatasourceSetup]]. # Suiviez le tutoriel [[ABCD2Mapping pour créer un mapping]] pour le schéma de données (par exemple DarwinCore ou ABCD) que vous voulez utiliser. # Testez et debuggez votre web service BioCASe. Lisez le tutoriel [[Debugging]] pour plus d'informations à ce sujet. # Quand vous avez terminé avec l'installation et la configuration du BioCASE Provider Software, vous pouvez enregistrer le wen service BioCASe auprès du réseau sur lequel vous souhaitez publiser vos données, par exemple le [http://www.gbif.org GBIF]. Nous vous recommandons de contacter l'[[Special:Contact|équipe BioCASe]] pour vérifier au préalable votre web service nous pourrons alors diagnistiquer des problèmes plus compliqué à résoudre après cet enregistrement. == Support == En cas de difficultés lors de l'installation et de la configuration, sous devriez: * jeter un oeil sur le [[FAQ]] qui contient peut-être une réponse à votre question * demander l'aide au staff technique d'un projet lié à BioCASe (comme CABIN ou OpenUp!) * demander l'aide d'autres utilisateurs de BioCASe * contacter l'[[Special:Contact|BioCASe team]], qui est plutôt anglophone. Décrivez le plus précisément possible votre environnement (système d'exploitation et système de bases de données utlisés), l'étape où le problème apparaît, et tous les messages d'erreurs ou informations techniques retourné par le logiciel. Il est recommendé d'envoyer des copies d'écran à vos interlocuteurs pour ce type de problème. b5c105a794add2107fdae6e4812cfec119a2ba2f 501 500 2012-04-10T15:20:08Z FranckTheeten 15 wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' == Qu'est ce que le Logiciel Fournisseur de données BioCASe? == Le logiciel fournisseur de données BioCASe (abrégé '''BPS'' pour '''BioCASe Provider Software''' en anglais ) est un middleware, c'est à dire un logiciel situéà une position intermédiaire entre une base de données et une interface graphique. Il produit des données en XML, afin de les publier sur un réseau d'information scientifique. Après l'installation de BioCASe et sa configuration pour un type de base de données particulier, l'information publiée sera accessible sous la forme d'un web service de type BioCASe. Cela veut dire que l'information pourra être interrogée et recherchée avec des requêtes au format BioCASe, car BioCASe désigne également un protocole XML pour envoyer des messages à un web service. L'intérêt de ce type de logiciel est qu'il est "agnostique" par rapport au modèle de données de la base de données d'origine, c'est à dire que les requêtes envoyées au web services n'ont pas besoin de connaître la structure de la base de données pour fonctionner. Il peut être également utilisé avec n'importe quel type de schéma conceptuel de données en XML. Le BPS est un ensemble d'outils qui doivent être installée sur un serveur Web installé chez le fournisseur de données. Son composant central est le '''PyWrapper''', une interface XML/CGI, écrite en Python, pour se connecter à des bases de données . Il permet un accès standardisé (c'est à dire via des procédures uniformes qui peuvent être ainsi partagée par tous les acteurs d'un réseau d'information) à une large gamme de systèmes de base de données dont la structure est arbitraire (c'est à dire définie par le fournisseur de données seul). Un certain nombre d'outils est réuni autour du BPS pour configurer, tester et débugger une installation ou un web service BioCASe. Même si BioCASe peut être utilisé avec n'importe quel schéma conceptuel défini en XML, son domaine de prédilection est la publication de données d'occurrence provenant de bases de données contenant des spécimens ou des observations sur des réseaux d'information primaire sur la biodiversité, comme le [http://www.biocase.org réseau BioCASe] et le [http://www.gbif.org GBIF (Global Biodiversity Information Facility)]. == Prérequis pour utiliser le BPS == Avant d'utiliser le logiciel fournisseur de données BioCASe, trois conditions doivent être vérifiées: * D'une part, les données à publier doivent être stockées dans une système gestionnaires de base de données relationnelle (parfois abrégé '''SGBDR''' ou '''RDBMS''' en anglais) compatible avec le langage '''SQL''', et dont le modèle de données est bien documenté, ou, au minimum, présent à l'esprit d'un gestionnaire de données. Les bases de donénes non compatibles avec SQL ne sont pas supportées par BioCASe. Actuellement, le logiciel peut se connecter au bases de données suivantes: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * Pour installer BioCASe, vous devez avoir un logiciel de serveur Web (Apache ou Microsoft IIS) accessible en permenante depuis l'Internet. Cela peut être le serveur d'une institution académique ou scientifique, ou un service d'hébérgement (même s'il est parfois pénible dans le second cas d'obtenir les droits d'installation auprès d'un service d'hébergement. Une connection commutée (peer to peer ou 'Dial-up') ne fonctionnera pas. Notez qu'il est tout à fait possuble à des fins d'entraînement, de formation ou de développment, d'installer un serveur Internet sur votre propre machine, isolé du reste de l'Internet, et de faire fonctionner BiocASe dessus. Il sera alors accessible localement pour l'ordinateur seum (via le nom http://localhost ou l'adresse IP réservée 127.0.0.1) * Vous devez avoir la possibilité d'installer Python sur le serveur web en question (si jamais ce n'est pas le cas) ainsi que des libraires ou paquets complémentaires Python. Cela peut être fait par un informaticien ou quelqu'un ayant de bonnes connaissances techniques, mais assurez vous que cela ne pose pas problèmes par rapport aux procédures et règles techniques qui pourraient être en vigeur dans votre institution. == Etapes == Une fois que vous vous êtes assurez que le BioCASe Provider Software est un choix pertinent pour vos besoins ([[Special:Contact|contactez l'équipe BioCASe]] si vous avez encore besoin d'information avant de prendre une décicion), vous devez exécutez les étapes suivantes: # Pensez à l'information que vous souhaitez publier sur le réseau. Cela implique d'avoir une réponse à la question de la propriété des données et des termes d'utlsiation, licences, copyright et restrictions dans l'utlisation qui valent pour les données publiées. Vous devriez aussi vous demander si certaines information ne devraient '''pas''' être publiées car cela pourrait avoir une conséquence sur des espèces menacées (une solution possible est de soustraire certaines occurrence du jeu de données publier ou de rendre moins précise une information exacte sur la localité d'observation). # Préparez la base de données comme décrit dand le tutoriel [[Preparation]]. Il faut alors décider de publier soit directement la base de données principale qui est utlisée pour les mises à jour, soit un 'snapshot' (une copie de tout ou une partie des données prise à un moment précis), de créer uen table de métadonnées (informations sur la base de donénes elle-même) et, dans certains cas, de transformer la présentation des données. # Suivez le tutoriel d'[[Installation]] pour installer le Fournisseur de données BioCASe. # Configurez un jeu de données BioCASe dans le logiciel et connectez-le à votre base de données. Configuez alors le jeu de données comme décrit dans le tutoriel [[DatasourceSetup]]. # Suiviez le tutoriel [[ABCD2Mapping pour créer un mapping]] pour le schéma de données (par exemple DarwinCore ou ABCD) que vous voulez utiliser. # Testez et debuggez votre web service BioCASe. Lisez le tutoriel [[Debugging]] pour plus d'informations à ce sujet. # Quand vous avez terminé avec l'installation et la configuration du BioCASE Provider Software, vous pouvez enregistrer le wen service BioCASe auprès du réseau sur lequel vous souhaitez publiser vos données, par exemple le [http://www.gbif.org GBIF]. Nous vous recommandons de contacter l'[[Special:Contact|équipe BioCASe]] pour vérifier au préalable votre web service nous pourrons alors diagnistiquer des problèmes plus compliqué à résoudre après cet enregistrement. == Support == En cas de difficultés lors de l'installation et de la configuration, sous devriez: * jeter un oeil sur le [[FAQ]] qui contient peut-être une réponse à votre question * demander l'aide au staff technique d'un projet lié à BioCASe (comme CABIN ou OpenUp!) * demander l'aide d'autres utilisateurs de BioCASe * contacter l'[[Special:Contact|BioCASe team]], qui est plutôt anglophone. Décrivez le plus précisément possible votre environnement (système d'exploitation et système de bases de données utlisés), l'étape où le problème apparaît, et tous les messages d'erreurs ou informations techniques retourné par le logiciel. Il est recommendé d'envoyer des copies d'écran à vos interlocuteurs pour ce type de problème. 973ad2f0d51c9df3ba1dbc516b38dac15c16a064 502 501 2012-04-10T15:21:50Z FranckTheeten 15 /* Prérequis pour utiliser le BPS */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' == Qu'est ce que le Logiciel Fournisseur de données BioCASe? == Le logiciel fournisseur de données BioCASe (abrégé '''BPS'' pour '''BioCASe Provider Software''' en anglais ) est un middleware, c'est à dire un logiciel situéà une position intermédiaire entre une base de données et une interface graphique. Il produit des données en XML, afin de les publier sur un réseau d'information scientifique. Après l'installation de BioCASe et sa configuration pour un type de base de données particulier, l'information publiée sera accessible sous la forme d'un web service de type BioCASe. Cela veut dire que l'information pourra être interrogée et recherchée avec des requêtes au format BioCASe, car BioCASe désigne également un protocole XML pour envoyer des messages à un web service. L'intérêt de ce type de logiciel est qu'il est "agnostique" par rapport au modèle de données de la base de données d'origine, c'est à dire que les requêtes envoyées au web services n'ont pas besoin de connaître la structure de la base de données pour fonctionner. Il peut être également utilisé avec n'importe quel type de schéma conceptuel de données en XML. Le BPS est un ensemble d'outils qui doivent être installée sur un serveur Web installé chez le fournisseur de données. Son composant central est le '''PyWrapper''', une interface XML/CGI, écrite en Python, pour se connecter à des bases de données . Il permet un accès standardisé (c'est à dire via des procédures uniformes qui peuvent être ainsi partagée par tous les acteurs d'un réseau d'information) à une large gamme de systèmes de base de données dont la structure est arbitraire (c'est à dire définie par le fournisseur de données seul). Un certain nombre d'outils est réuni autour du BPS pour configurer, tester et débugger une installation ou un web service BioCASe. Même si BioCASe peut être utilisé avec n'importe quel schéma conceptuel défini en XML, son domaine de prédilection est la publication de données d'occurrence provenant de bases de données contenant des spécimens ou des observations sur des réseaux d'information primaire sur la biodiversité, comme le [http://www.biocase.org réseau BioCASe] et le [http://www.gbif.org GBIF (Global Biodiversity Information Facility)]. == Prérequis pour utiliser le BPS == Avant d'utiliser le logiciel fournisseur de données BioCASe, trois conditions doivent être vérifiées: * D'une part, les données à publier doivent être stockées dans une système gestionnaires de base de données relationnelles (parfois abrégé '''SGBDR''' ou '''RDBMS''' en anglais) compatible avec le langage '''SQL'''. Son modèle de données doit être bien documenté, ou, au minimum, présent à l'esprit d'un gestionnaire de données. Les bases de données non compatibles avec SQL ne sont pas supportées par BioCASe. Actuellement, le logiciel peut se connecter au bases de données suivantes: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * Pour installer BioCASe, vous devez avoir un logiciel de serveur Web (Apache ou Microsoft IIS) accessible en permenante depuis l'Internet. Cela peut être le serveur d'une institution académique ou scientifique, ou un service d'hébérgement (même s'il est parfois pénible dans le second cas d'obtenir les droits d'installation auprès d'un service d'hébergement. Une connection commutée (peer to peer ou 'Dial-up') ne fonctionnera pas. Notez qu'il est tout à fait possuble à des fins d'entraînement, de formation ou de développment, d'installer un serveur Internet sur votre propre machine, isolé du reste de l'Internet, et de faire fonctionner BiocASe dessus. Il sera alors accessible localement pour l'ordinateur seum (via le nom http://localhost ou l'adresse IP réservée 127.0.0.1) * Vous devez avoir la possibilité d'installer Python sur le serveur web en question (si jamais ce n'est pas le cas) ainsi que des libraires ou paquets complémentaires Python. Cela peut être fait par un informaticien ou quelqu'un ayant de bonnes connaissances techniques, mais assurez vous que cela ne pose pas problèmes par rapport aux procédures et règles techniques qui pourraient être en vigeur dans votre institution. == Etapes == Une fois que vous vous êtes assurez que le BioCASe Provider Software est un choix pertinent pour vos besoins ([[Special:Contact|contactez l'équipe BioCASe]] si vous avez encore besoin d'information avant de prendre une décicion), vous devez exécutez les étapes suivantes: # Pensez à l'information que vous souhaitez publier sur le réseau. Cela implique d'avoir une réponse à la question de la propriété des données et des termes d'utlsiation, licences, copyright et restrictions dans l'utlisation qui valent pour les données publiées. Vous devriez aussi vous demander si certaines information ne devraient '''pas''' être publiées car cela pourrait avoir une conséquence sur des espèces menacées (une solution possible est de soustraire certaines occurrence du jeu de données publier ou de rendre moins précise une information exacte sur la localité d'observation). # Préparez la base de données comme décrit dand le tutoriel [[Preparation]]. Il faut alors décider de publier soit directement la base de données principale qui est utlisée pour les mises à jour, soit un 'snapshot' (une copie de tout ou une partie des données prise à un moment précis), de créer uen table de métadonnées (informations sur la base de donénes elle-même) et, dans certains cas, de transformer la présentation des données. # Suivez le tutoriel d'[[Installation]] pour installer le Fournisseur de données BioCASe. # Configurez un jeu de données BioCASe dans le logiciel et connectez-le à votre base de données. Configuez alors le jeu de données comme décrit dans le tutoriel [[DatasourceSetup]]. # Suiviez le tutoriel [[ABCD2Mapping pour créer un mapping]] pour le schéma de données (par exemple DarwinCore ou ABCD) que vous voulez utiliser. # Testez et debuggez votre web service BioCASe. Lisez le tutoriel [[Debugging]] pour plus d'informations à ce sujet. # Quand vous avez terminé avec l'installation et la configuration du BioCASE Provider Software, vous pouvez enregistrer le wen service BioCASe auprès du réseau sur lequel vous souhaitez publiser vos données, par exemple le [http://www.gbif.org GBIF]. Nous vous recommandons de contacter l'[[Special:Contact|équipe BioCASe]] pour vérifier au préalable votre web service nous pourrons alors diagnistiquer des problèmes plus compliqué à résoudre après cet enregistrement. == Support == En cas de difficultés lors de l'installation et de la configuration, sous devriez: * jeter un oeil sur le [[FAQ]] qui contient peut-être une réponse à votre question * demander l'aide au staff technique d'un projet lié à BioCASe (comme CABIN ou OpenUp!) * demander l'aide d'autres utilisateurs de BioCASe * contacter l'[[Special:Contact|BioCASe team]], qui est plutôt anglophone. Décrivez le plus précisément possible votre environnement (système d'exploitation et système de bases de données utlisés), l'étape où le problème apparaît, et tous les messages d'erreurs ou informations techniques retourné par le logiciel. Il est recommendé d'envoyer des copies d'écran à vos interlocuteurs pour ce type de problème. c98659beda65595d54804e273ea89cfe5c016689 503 502 2012-04-10T15:26:28Z FranckTheeten 15 /* Prérequis pour utiliser le BPS */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' == Qu'est ce que le Logiciel Fournisseur de données BioCASe? == Le logiciel fournisseur de données BioCASe (abrégé '''BPS'' pour '''BioCASe Provider Software''' en anglais ) est un middleware, c'est à dire un logiciel situéà une position intermédiaire entre une base de données et une interface graphique. Il produit des données en XML, afin de les publier sur un réseau d'information scientifique. Après l'installation de BioCASe et sa configuration pour un type de base de données particulier, l'information publiée sera accessible sous la forme d'un web service de type BioCASe. Cela veut dire que l'information pourra être interrogée et recherchée avec des requêtes au format BioCASe, car BioCASe désigne également un protocole XML pour envoyer des messages à un web service. L'intérêt de ce type de logiciel est qu'il est "agnostique" par rapport au modèle de données de la base de données d'origine, c'est à dire que les requêtes envoyées au web services n'ont pas besoin de connaître la structure de la base de données pour fonctionner. Il peut être également utilisé avec n'importe quel type de schéma conceptuel de données en XML. Le BPS est un ensemble d'outils qui doivent être installée sur un serveur Web installé chez le fournisseur de données. Son composant central est le '''PyWrapper''', une interface XML/CGI, écrite en Python, pour se connecter à des bases de données . Il permet un accès standardisé (c'est à dire via des procédures uniformes qui peuvent être ainsi partagée par tous les acteurs d'un réseau d'information) à une large gamme de systèmes de base de données dont la structure est arbitraire (c'est à dire définie par le fournisseur de données seul). Un certain nombre d'outils est réuni autour du BPS pour configurer, tester et débugger une installation ou un web service BioCASe. Même si BioCASe peut être utilisé avec n'importe quel schéma conceptuel défini en XML, son domaine de prédilection est la publication de données d'occurrence provenant de bases de données contenant des spécimens ou des observations sur des réseaux d'information primaire sur la biodiversité, comme le [http://www.biocase.org réseau BioCASe] et le [http://www.gbif.org GBIF (Global Biodiversity Information Facility)]. == Prérequis pour utiliser le BPS == Avant d'utiliser le logiciel fournisseur de données BioCASe, trois conditions doivent être vérifiées: * D'une part, les données à publier doivent être stockées dans une système gestionnaires de base de données relationnelles (parfois abrégé '''SGBDR''' ou '''RDBMS''' en anglais) compatible avec le langage '''SQL'''. Son modèle de données doit être bien documenté, ou, au minimum, présent à l'esprit d'un gestionnaire de données. Les bases de données non compatibles avec SQL ne sont pas supportées par BioCASe. Actuellement, le logiciel peut se connecter au bases de données suivantes: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * Pour installer BioCASe, vous devez avoir un logiciel de serveur Web (Apache ou Microsoft IIS) accessible en permanence depuis l'Internet. Cela peut être le serveur d'une institution académique ou scientifique, ou un service d'hébergement (il est parfois pénible dans le second cas d'obtenir les droits d'installation auprès d'un service d'hébergement). Une connexion commutée (peer to peer ou 'Dial-up') ne fonctionnera pas. Notez qu'il est tout à fait possible d'installer un serveur Internet sur votre propre machine, isolé du reste de l'Internet, et de faire fonctionner BioCASe dessus à des fins d'entraînement, de formation ou de développement, . Il sera alors accessible localement pour l'ordinateur seul (via le nom http://localhost ou l'adresse IP réservée 127.0.0.1) * Vous devez avoir la possibilité d'installer Python sur le serveur web en question (si jamais ce n'est pas le cas) ainsi que des librairies ou paquets complémentaires Python. Cela peut être fait par un informaticien ou quelqu'un ayant de bonnes connaissances techniques, mais assurez vous que cela ne pose pas problèmes par rapport aux procédures et règles techniques qui pourraient être en vigueur dans votre institution. == Etapes == Une fois que vous vous êtes assurez que le BioCASe Provider Software est un choix pertinent pour vos besoins ([[Special:Contact|contactez l'équipe BioCASe]] si vous avez encore besoin d'information avant de prendre une décicion), vous devez exécutez les étapes suivantes: # Pensez à l'information que vous souhaitez publier sur le réseau. Cela implique d'avoir une réponse à la question de la propriété des données et des termes d'utlsiation, licences, copyright et restrictions dans l'utlisation qui valent pour les données publiées. Vous devriez aussi vous demander si certaines information ne devraient '''pas''' être publiées car cela pourrait avoir une conséquence sur des espèces menacées (une solution possible est de soustraire certaines occurrence du jeu de données publier ou de rendre moins précise une information exacte sur la localité d'observation). # Préparez la base de données comme décrit dand le tutoriel [[Preparation]]. Il faut alors décider de publier soit directement la base de données principale qui est utlisée pour les mises à jour, soit un 'snapshot' (une copie de tout ou une partie des données prise à un moment précis), de créer uen table de métadonnées (informations sur la base de donénes elle-même) et, dans certains cas, de transformer la présentation des données. # Suivez le tutoriel d'[[Installation]] pour installer le Fournisseur de données BioCASe. # Configurez un jeu de données BioCASe dans le logiciel et connectez-le à votre base de données. Configuez alors le jeu de données comme décrit dans le tutoriel [[DatasourceSetup]]. # Suiviez le tutoriel [[ABCD2Mapping pour créer un mapping]] pour le schéma de données (par exemple DarwinCore ou ABCD) que vous voulez utiliser. # Testez et debuggez votre web service BioCASe. Lisez le tutoriel [[Debugging]] pour plus d'informations à ce sujet. # Quand vous avez terminé avec l'installation et la configuration du BioCASE Provider Software, vous pouvez enregistrer le wen service BioCASe auprès du réseau sur lequel vous souhaitez publiser vos données, par exemple le [http://www.gbif.org GBIF]. Nous vous recommandons de contacter l'[[Special:Contact|équipe BioCASe]] pour vérifier au préalable votre web service nous pourrons alors diagnistiquer des problèmes plus compliqué à résoudre après cet enregistrement. == Support == En cas de difficultés lors de l'installation et de la configuration, sous devriez: * jeter un oeil sur le [[FAQ]] qui contient peut-être une réponse à votre question * demander l'aide au staff technique d'un projet lié à BioCASe (comme CABIN ou OpenUp!) * demander l'aide d'autres utilisateurs de BioCASe * contacter l'[[Special:Contact|BioCASe team]], qui est plutôt anglophone. Décrivez le plus précisément possible votre environnement (système d'exploitation et système de bases de données utlisés), l'étape où le problème apparaît, et tous les messages d'erreurs ou informations techniques retourné par le logiciel. Il est recommendé d'envoyer des copies d'écran à vos interlocuteurs pour ce type de problème. fa660f38bcce53c4db95d0cd69292a3e881bfdc8 504 503 2012-04-10T15:26:51Z FranckTheeten 15 /* Prérequis pour utiliser le BPS */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' == Qu'est ce que le Logiciel Fournisseur de données BioCASe? == Le logiciel fournisseur de données BioCASe (abrégé '''BPS'' pour '''BioCASe Provider Software''' en anglais ) est un middleware, c'est à dire un logiciel situéà une position intermédiaire entre une base de données et une interface graphique. Il produit des données en XML, afin de les publier sur un réseau d'information scientifique. Après l'installation de BioCASe et sa configuration pour un type de base de données particulier, l'information publiée sera accessible sous la forme d'un web service de type BioCASe. Cela veut dire que l'information pourra être interrogée et recherchée avec des requêtes au format BioCASe, car BioCASe désigne également un protocole XML pour envoyer des messages à un web service. L'intérêt de ce type de logiciel est qu'il est "agnostique" par rapport au modèle de données de la base de données d'origine, c'est à dire que les requêtes envoyées au web services n'ont pas besoin de connaître la structure de la base de données pour fonctionner. Il peut être également utilisé avec n'importe quel type de schéma conceptuel de données en XML. Le BPS est un ensemble d'outils qui doivent être installée sur un serveur Web installé chez le fournisseur de données. Son composant central est le '''PyWrapper''', une interface XML/CGI, écrite en Python, pour se connecter à des bases de données . Il permet un accès standardisé (c'est à dire via des procédures uniformes qui peuvent être ainsi partagée par tous les acteurs d'un réseau d'information) à une large gamme de systèmes de base de données dont la structure est arbitraire (c'est à dire définie par le fournisseur de données seul). Un certain nombre d'outils est réuni autour du BPS pour configurer, tester et débugger une installation ou un web service BioCASe. Même si BioCASe peut être utilisé avec n'importe quel schéma conceptuel défini en XML, son domaine de prédilection est la publication de données d'occurrence provenant de bases de données contenant des spécimens ou des observations sur des réseaux d'information primaire sur la biodiversité, comme le [http://www.biocase.org réseau BioCASe] et le [http://www.gbif.org GBIF (Global Biodiversity Information Facility)]. == Prérequis pour utiliser le BPS == Avant d'utiliser le logiciel fournisseur de données BioCASe, trois conditions doivent être vérifiées: * D'une part, les données à publier doivent être stockées dans une système gestionnaires de base de données relationnelles (parfois abrégé '''SGBDR''' ou '''RDBMS''' en anglais) compatible avec le langage '''SQL'''. Son modèle de données doit être bien documenté, ou, au minimum, présent à l'esprit d'un gestionnaire de données. Les bases de données non compatibles avec SQL ne sont pas supportées par BioCASe. Actuellement, le logiciel peut se connecter au bases de données suivantes: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * Pour installer BioCASe, vous devez avoir un logiciel de serveur Web (Apache ou Microsoft IIS) accessible en permanence depuis l'Internet. Cela peut être le serveur d'une institution académique ou scientifique, ou un service d'hébergement (il est parfois pénible dans le second cas d'obtenir les droits d'installation auprès d'un service d'hébergement). Une connexion commutée (peer to peer ou 'Dial-up') ne fonctionnera pas. Notez qu'il est tout à fait possible d'installer un serveur Internet sur votre propre machine, isolé du reste de l'Internet, et de faire fonctionner BioCASe dessus à des fins d'entraînement, de formation ou de développement, . Il sera alors accessible localement pour l'ordinateur seul (via le nom http://localhost ou l'adresse IP réservée 127.0.0.1) * Vous devez avoir la possibilité d'installer Python sur le serveur web en question (si jamais ce n'est pas le cas) ainsi que des librairies ou paquets complémentaires Python. Cela peut être effectué relativement facilement par un informaticien ou quelqu'un ayant de bonnes connaissances techniques, mais assurez vous que cela ne pose pas problèmes par rapport aux procédures et règles techniques qui pourraient être en vigueur dans votre institution. == Etapes == Une fois que vous vous êtes assurez que le BioCASe Provider Software est un choix pertinent pour vos besoins ([[Special:Contact|contactez l'équipe BioCASe]] si vous avez encore besoin d'information avant de prendre une décicion), vous devez exécutez les étapes suivantes: # Pensez à l'information que vous souhaitez publier sur le réseau. Cela implique d'avoir une réponse à la question de la propriété des données et des termes d'utlsiation, licences, copyright et restrictions dans l'utlisation qui valent pour les données publiées. Vous devriez aussi vous demander si certaines information ne devraient '''pas''' être publiées car cela pourrait avoir une conséquence sur des espèces menacées (une solution possible est de soustraire certaines occurrence du jeu de données publier ou de rendre moins précise une information exacte sur la localité d'observation). # Préparez la base de données comme décrit dand le tutoriel [[Preparation]]. Il faut alors décider de publier soit directement la base de données principale qui est utlisée pour les mises à jour, soit un 'snapshot' (une copie de tout ou une partie des données prise à un moment précis), de créer uen table de métadonnées (informations sur la base de donénes elle-même) et, dans certains cas, de transformer la présentation des données. # Suivez le tutoriel d'[[Installation]] pour installer le Fournisseur de données BioCASe. # Configurez un jeu de données BioCASe dans le logiciel et connectez-le à votre base de données. Configuez alors le jeu de données comme décrit dans le tutoriel [[DatasourceSetup]]. # Suiviez le tutoriel [[ABCD2Mapping pour créer un mapping]] pour le schéma de données (par exemple DarwinCore ou ABCD) que vous voulez utiliser. # Testez et debuggez votre web service BioCASe. Lisez le tutoriel [[Debugging]] pour plus d'informations à ce sujet. # Quand vous avez terminé avec l'installation et la configuration du BioCASE Provider Software, vous pouvez enregistrer le wen service BioCASe auprès du réseau sur lequel vous souhaitez publiser vos données, par exemple le [http://www.gbif.org GBIF]. Nous vous recommandons de contacter l'[[Special:Contact|équipe BioCASe]] pour vérifier au préalable votre web service nous pourrons alors diagnistiquer des problèmes plus compliqué à résoudre après cet enregistrement. == Support == En cas de difficultés lors de l'installation et de la configuration, sous devriez: * jeter un oeil sur le [[FAQ]] qui contient peut-être une réponse à votre question * demander l'aide au staff technique d'un projet lié à BioCASe (comme CABIN ou OpenUp!) * demander l'aide d'autres utilisateurs de BioCASe * contacter l'[[Special:Contact|BioCASe team]], qui est plutôt anglophone. Décrivez le plus précisément possible votre environnement (système d'exploitation et système de bases de données utlisés), l'étape où le problème apparaît, et tous les messages d'erreurs ou informations techniques retourné par le logiciel. Il est recommendé d'envoyer des copies d'écran à vos interlocuteurs pour ce type de problème. 3a86e35d31ab9e35858325d92a6da2757415aaed 505 504 2012-04-10T15:28:22Z FranckTheeten 15 /* Etapes */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' == Qu'est ce que le Logiciel Fournisseur de données BioCASe? == Le logiciel fournisseur de données BioCASe (abrégé '''BPS'' pour '''BioCASe Provider Software''' en anglais ) est un middleware, c'est à dire un logiciel situéà une position intermédiaire entre une base de données et une interface graphique. Il produit des données en XML, afin de les publier sur un réseau d'information scientifique. Après l'installation de BioCASe et sa configuration pour un type de base de données particulier, l'information publiée sera accessible sous la forme d'un web service de type BioCASe. Cela veut dire que l'information pourra être interrogée et recherchée avec des requêtes au format BioCASe, car BioCASe désigne également un protocole XML pour envoyer des messages à un web service. L'intérêt de ce type de logiciel est qu'il est "agnostique" par rapport au modèle de données de la base de données d'origine, c'est à dire que les requêtes envoyées au web services n'ont pas besoin de connaître la structure de la base de données pour fonctionner. Il peut être également utilisé avec n'importe quel type de schéma conceptuel de données en XML. Le BPS est un ensemble d'outils qui doivent être installée sur un serveur Web installé chez le fournisseur de données. Son composant central est le '''PyWrapper''', une interface XML/CGI, écrite en Python, pour se connecter à des bases de données . Il permet un accès standardisé (c'est à dire via des procédures uniformes qui peuvent être ainsi partagée par tous les acteurs d'un réseau d'information) à une large gamme de systèmes de base de données dont la structure est arbitraire (c'est à dire définie par le fournisseur de données seul). Un certain nombre d'outils est réuni autour du BPS pour configurer, tester et débugger une installation ou un web service BioCASe. Même si BioCASe peut être utilisé avec n'importe quel schéma conceptuel défini en XML, son domaine de prédilection est la publication de données d'occurrence provenant de bases de données contenant des spécimens ou des observations sur des réseaux d'information primaire sur la biodiversité, comme le [http://www.biocase.org réseau BioCASe] et le [http://www.gbif.org GBIF (Global Biodiversity Information Facility)]. == Prérequis pour utiliser le BPS == Avant d'utiliser le logiciel fournisseur de données BioCASe, trois conditions doivent être vérifiées: * D'une part, les données à publier doivent être stockées dans une système gestionnaires de base de données relationnelles (parfois abrégé '''SGBDR''' ou '''RDBMS''' en anglais) compatible avec le langage '''SQL'''. Son modèle de données doit être bien documenté, ou, au minimum, présent à l'esprit d'un gestionnaire de données. Les bases de données non compatibles avec SQL ne sont pas supportées par BioCASe. Actuellement, le logiciel peut se connecter au bases de données suivantes: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * Pour installer BioCASe, vous devez avoir un logiciel de serveur Web (Apache ou Microsoft IIS) accessible en permanence depuis l'Internet. Cela peut être le serveur d'une institution académique ou scientifique, ou un service d'hébergement (il est parfois pénible dans le second cas d'obtenir les droits d'installation auprès d'un service d'hébergement). Une connexion commutée (peer to peer ou 'Dial-up') ne fonctionnera pas. Notez qu'il est tout à fait possible d'installer un serveur Internet sur votre propre machine, isolé du reste de l'Internet, et de faire fonctionner BioCASe dessus à des fins d'entraînement, de formation ou de développement, . Il sera alors accessible localement pour l'ordinateur seul (via le nom http://localhost ou l'adresse IP réservée 127.0.0.1) * Vous devez avoir la possibilité d'installer Python sur le serveur web en question (si jamais ce n'est pas le cas) ainsi que des librairies ou paquets complémentaires Python. Cela peut être effectué relativement facilement par un informaticien ou quelqu'un ayant de bonnes connaissances techniques, mais assurez vous que cela ne pose pas problèmes par rapport aux procédures et règles techniques qui pourraient être en vigueur dans votre institution. == Etapes == Une fois que vous vous êtes assurez que le BioCASe Provider Software est un choix pertinent pour vos besoins ([[Special:Contact|contactez l'équipe BioCASe]] si vous souhaitez plus d'information avant de prendre une décision), vous devez exécuter les étapes suivantes: # Pensez à l'information que vous souhaitez publier sur le réseau. Cela implique d'avoir réfléchi et répondu à la question de la propriété des données et des termes d'utilisation, licences, copyright et restrictions dans l'utilisation qui s'appliquent à leur publication. Vous devriez aussi vous demander si certaines information ne devraient '''pas''' être publiées car cela pourrait avoir une conséquence sur des espèces menacées (une solution possible est de soustraire certaines occurrence du jeu de données publier ou de rendre moins précise une information exacte sur la localité d'observation). # Préparez la base de données comme décrit dand le tutoriel [[Preparation]]. Il faut alors décider de publier soit directement la base de données principale qui est utlisée pour les mises à jour, soit un 'snapshot' (une copie de tout ou une partie des données prise à un moment précis), de créer uen table de métadonnées (informations sur la base de donénes elle-même) et, dans certains cas, de transformer la présentation des données. # Suivez le tutoriel d'[[Installation]] pour installer le Fournisseur de données BioCASe. # Configurez un jeu de données BioCASe dans le logiciel et connectez-le à votre base de données. Configuez alors le jeu de données comme décrit dans le tutoriel [[DatasourceSetup]]. # Suiviez le tutoriel [[ABCD2Mapping pour créer un mapping]] pour le schéma de données (par exemple DarwinCore ou ABCD) que vous voulez utiliser. # Testez et debuggez votre web service BioCASe. Lisez le tutoriel [[Debugging]] pour plus d'informations à ce sujet. # Quand vous avez terminé avec l'installation et la configuration du BioCASE Provider Software, vous pouvez enregistrer le wen service BioCASe auprès du réseau sur lequel vous souhaitez publiser vos données, par exemple le [http://www.gbif.org GBIF]. Nous vous recommandons de contacter l'[[Special:Contact|équipe BioCASe]] pour vérifier au préalable votre web service nous pourrons alors diagnistiquer des problèmes plus compliqué à résoudre après cet enregistrement. == Support == En cas de difficultés lors de l'installation et de la configuration, sous devriez: * jeter un oeil sur le [[FAQ]] qui contient peut-être une réponse à votre question * demander l'aide au staff technique d'un projet lié à BioCASe (comme CABIN ou OpenUp!) * demander l'aide d'autres utilisateurs de BioCASe * contacter l'[[Special:Contact|BioCASe team]], qui est plutôt anglophone. Décrivez le plus précisément possible votre environnement (système d'exploitation et système de bases de données utlisés), l'étape où le problème apparaît, et tous les messages d'erreurs ou informations techniques retourné par le logiciel. Il est recommendé d'envoyer des copies d'écran à vos interlocuteurs pour ce type de problème. 354a8d09fb9ad135421c46cb912ec0d541747584 506 505 2012-04-10T15:29:53Z FranckTheeten 15 /* Etapes */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' == Qu'est ce que le Logiciel Fournisseur de données BioCASe? == Le logiciel fournisseur de données BioCASe (abrégé '''BPS'' pour '''BioCASe Provider Software''' en anglais ) est un middleware, c'est à dire un logiciel situéà une position intermédiaire entre une base de données et une interface graphique. Il produit des données en XML, afin de les publier sur un réseau d'information scientifique. Après l'installation de BioCASe et sa configuration pour un type de base de données particulier, l'information publiée sera accessible sous la forme d'un web service de type BioCASe. Cela veut dire que l'information pourra être interrogée et recherchée avec des requêtes au format BioCASe, car BioCASe désigne également un protocole XML pour envoyer des messages à un web service. L'intérêt de ce type de logiciel est qu'il est "agnostique" par rapport au modèle de données de la base de données d'origine, c'est à dire que les requêtes envoyées au web services n'ont pas besoin de connaître la structure de la base de données pour fonctionner. Il peut être également utilisé avec n'importe quel type de schéma conceptuel de données en XML. Le BPS est un ensemble d'outils qui doivent être installée sur un serveur Web installé chez le fournisseur de données. Son composant central est le '''PyWrapper''', une interface XML/CGI, écrite en Python, pour se connecter à des bases de données . Il permet un accès standardisé (c'est à dire via des procédures uniformes qui peuvent être ainsi partagée par tous les acteurs d'un réseau d'information) à une large gamme de systèmes de base de données dont la structure est arbitraire (c'est à dire définie par le fournisseur de données seul). Un certain nombre d'outils est réuni autour du BPS pour configurer, tester et débugger une installation ou un web service BioCASe. Même si BioCASe peut être utilisé avec n'importe quel schéma conceptuel défini en XML, son domaine de prédilection est la publication de données d'occurrence provenant de bases de données contenant des spécimens ou des observations sur des réseaux d'information primaire sur la biodiversité, comme le [http://www.biocase.org réseau BioCASe] et le [http://www.gbif.org GBIF (Global Biodiversity Information Facility)]. == Prérequis pour utiliser le BPS == Avant d'utiliser le logiciel fournisseur de données BioCASe, trois conditions doivent être vérifiées: * D'une part, les données à publier doivent être stockées dans une système gestionnaires de base de données relationnelles (parfois abrégé '''SGBDR''' ou '''RDBMS''' en anglais) compatible avec le langage '''SQL'''. Son modèle de données doit être bien documenté, ou, au minimum, présent à l'esprit d'un gestionnaire de données. Les bases de données non compatibles avec SQL ne sont pas supportées par BioCASe. Actuellement, le logiciel peut se connecter au bases de données suivantes: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * Pour installer BioCASe, vous devez avoir un logiciel de serveur Web (Apache ou Microsoft IIS) accessible en permanence depuis l'Internet. Cela peut être le serveur d'une institution académique ou scientifique, ou un service d'hébergement (il est parfois pénible dans le second cas d'obtenir les droits d'installation auprès d'un service d'hébergement). Une connexion commutée (peer to peer ou 'Dial-up') ne fonctionnera pas. Notez qu'il est tout à fait possible d'installer un serveur Internet sur votre propre machine, isolé du reste de l'Internet, et de faire fonctionner BioCASe dessus à des fins d'entraînement, de formation ou de développement, . Il sera alors accessible localement pour l'ordinateur seul (via le nom http://localhost ou l'adresse IP réservée 127.0.0.1) * Vous devez avoir la possibilité d'installer Python sur le serveur web en question (si jamais ce n'est pas le cas) ainsi que des librairies ou paquets complémentaires Python. Cela peut être effectué relativement facilement par un informaticien ou quelqu'un ayant de bonnes connaissances techniques, mais assurez vous que cela ne pose pas problèmes par rapport aux procédures et règles techniques qui pourraient être en vigueur dans votre institution. == Etapes == Une fois que vous vous êtes assurez que le BioCASe Provider Software est un choix pertinent pour vos besoins ([[Special:Contact|contactez l'équipe BioCASe]] si vous souhaitez plus d'information avant de prendre une décision), vous devez exécuter les étapes suivantes: # Pensez à l'information que vous souhaitez publier sur le réseau. Cela implique d'avoir réfléchi et répondu à la question de la propriété des données et des termes d'utilisation, licences, copyright et restrictions dans l'utilisation qui s'appliquent à leur publication. Vous devriez aussi vous demander si certaines information ne devraient '''pas''' être publiées car cela pourrait avoir une conséquence sur des espèces menacées. une solution possible est de soustraire certaines occurrence du jeu de données publier, ou de rendre moins précise une information exacte sur la localité d'observation. # Préparez la base de données comme décrit dans le tutoriel [[Preparation]]. C'est à cette étape que doit être prise la décision de publier soit directement la base de données principale qui est utilisée pour les mises à jour, soit un 'snapshot' (c'est à dire une copie de tout ou une partie des données prise à un moment précis), de créer une table de métadonnées (informations sur la base de données elle-même) et, dans certains cas, de transformer la présentation des données. # Suivez le tutoriel d'[[Installation]] pour installer le Fournisseur de données BioCASe. # Configurez un jeu de données BioCASe dans le logiciel et connectez-le à votre base de données. Configuez alors le jeu de données comme décrit dans le tutoriel [[DatasourceSetup]]. # Suiviez le tutoriel [[ABCD2Mapping pour créer un mapping]] pour le schéma de données (par exemple DarwinCore ou ABCD) que vous voulez utiliser. # Testez et débuggez votre web service BioCASe. Lisez le tutoriel [[Debugging]] pour plus d'informations à ce sujet. # Quand vous avez terminé avec l'installation et la configuration du BioCASE Provider Software, vous pouvez enregistrer le wen service BioCASe auprès du réseau sur lequel vous souhaitez publiser vos données, par exemple le [http://www.gbif.org GBIF]. Nous vous recommandons de contacter l'[[Special:Contact|équipe BioCASe]] pour vérifier au préalable votre web service nous pourrons alors diagnistiquer des problèmes plus compliqué à résoudre après cet enregistrement. == Support == En cas de difficultés lors de l'installation et de la configuration, sous devriez: * jeter un oeil sur le [[FAQ]] qui contient peut-être une réponse à votre question * demander l'aide au staff technique d'un projet lié à BioCASe (comme CABIN ou OpenUp!) * demander l'aide d'autres utilisateurs de BioCASe * contacter l'[[Special:Contact|BioCASe team]], qui est plutôt anglophone. Décrivez le plus précisément possible votre environnement (système d'exploitation et système de bases de données utlisés), l'étape où le problème apparaît, et tous les messages d'erreurs ou informations techniques retourné par le logiciel. Il est recommendé d'envoyer des copies d'écran à vos interlocuteurs pour ce type de problème. ad20b85e694d64fe8a539d6fe601b66f88ad2efa 507 506 2012-04-10T15:30:44Z FranckTheeten 15 /* Prérequis pour utiliser le BPS */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' == Qu'est ce que le Logiciel Fournisseur de données BioCASe? == Le logiciel fournisseur de données BioCASe (abrégé '''BPS'' pour '''BioCASe Provider Software''' en anglais ) est un middleware, c'est à dire un logiciel situéà une position intermédiaire entre une base de données et une interface graphique. Il produit des données en XML, afin de les publier sur un réseau d'information scientifique. Après l'installation de BioCASe et sa configuration pour un type de base de données particulier, l'information publiée sera accessible sous la forme d'un web service de type BioCASe. Cela veut dire que l'information pourra être interrogée et recherchée avec des requêtes au format BioCASe, car BioCASe désigne également un protocole XML pour envoyer des messages à un web service. L'intérêt de ce type de logiciel est qu'il est "agnostique" par rapport au modèle de données de la base de données d'origine, c'est à dire que les requêtes envoyées au web services n'ont pas besoin de connaître la structure de la base de données pour fonctionner. Il peut être également utilisé avec n'importe quel type de schéma conceptuel de données en XML. Le BPS est un ensemble d'outils qui doivent être installée sur un serveur Web installé chez le fournisseur de données. Son composant central est le '''PyWrapper''', une interface XML/CGI, écrite en Python, pour se connecter à des bases de données . Il permet un accès standardisé (c'est à dire via des procédures uniformes qui peuvent être ainsi partagée par tous les acteurs d'un réseau d'information) à une large gamme de systèmes de base de données dont la structure est arbitraire (c'est à dire définie par le fournisseur de données seul). Un certain nombre d'outils est réuni autour du BPS pour configurer, tester et débugger une installation ou un web service BioCASe. Même si BioCASe peut être utilisé avec n'importe quel schéma conceptuel défini en XML, son domaine de prédilection est la publication de données d'occurrence provenant de bases de données contenant des spécimens ou des observations sur des réseaux d'information primaire sur la biodiversité, comme le [http://www.biocase.org réseau BioCASe] et le [http://www.gbif.org GBIF (Global Biodiversity Information Facility)]. == Prérequis pour utiliser le BPS == Avant d'utiliser le logiciel fournisseur de données BioCASe, trois conditions doivent être vérifiées: * D'une part, les données à publier doivent être stockées dans une système gestionnaires de base de données relationnelles (parfois abrégé '''SGBDR''' ou '''RDBMS''' en anglais) compatible avec le langage '''SQL'''. Son modèle de données doit être bien documenté, ou, au minimum, présent à l'esprit d'un gestionnaire de données. Les bases de données non compatibles avec SQL ne sont pas supportées par BioCASe. Actuellement, le logiciel peut se connecter au bases de données suivantes: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * Pour installer BioCASe, vous devez avoir un logiciel de serveur Web (Apache ou Microsoft IIS) accessible en permanence depuis l'Internet. Cela peut être le serveur d'une institution académique ou scientifique, ou un service d'hébergement. Il est parfois pénible dans ce second cas d'obtenir les droits d'installation auprès d'un service d'hébergement. Une connexion commutée peer to peer ou 'Dial-up' ne fonctionnera pas. Notez qu'il est tout à fait possible d'installer un serveur Internet sur votre propre machine, isolé du reste de l'Internet, et de faire fonctionner BioCASe dessus à des fins d'entraînement, de formation ou de développement, . Il sera alors accessible localement pour l'ordinateur seul (via le nom http://localhost ou l'adresse IP réservée 127.0.0.1) * Vous devez avoir la possibilité d'installer Python sur le serveur web en question (si jamais ce n'est pas le cas) ainsi que des librairies ou paquets complémentaires Python. Cela peut être effectué relativement facilement par un informaticien ou quelqu'un ayant de bonnes connaissances techniques, mais assurez vous que cela ne pose pas problèmes par rapport aux procédures et règles techniques qui pourraient être en vigueur dans votre institution. == Etapes == Une fois que vous vous êtes assurez que le BioCASe Provider Software est un choix pertinent pour vos besoins ([[Special:Contact|contactez l'équipe BioCASe]] si vous souhaitez plus d'information avant de prendre une décision), vous devez exécuter les étapes suivantes: # Pensez à l'information que vous souhaitez publier sur le réseau. Cela implique d'avoir réfléchi et répondu à la question de la propriété des données et des termes d'utilisation, licences, copyright et restrictions dans l'utilisation qui s'appliquent à leur publication. Vous devriez aussi vous demander si certaines information ne devraient '''pas''' être publiées car cela pourrait avoir une conséquence sur des espèces menacées. une solution possible est de soustraire certaines occurrence du jeu de données publier, ou de rendre moins précise une information exacte sur la localité d'observation. # Préparez la base de données comme décrit dans le tutoriel [[Preparation]]. C'est à cette étape que doit être prise la décision de publier soit directement la base de données principale qui est utilisée pour les mises à jour, soit un 'snapshot' (c'est à dire une copie de tout ou une partie des données prise à un moment précis), de créer une table de métadonnées (informations sur la base de données elle-même) et, dans certains cas, de transformer la présentation des données. # Suivez le tutoriel d'[[Installation]] pour installer le Fournisseur de données BioCASe. # Configurez un jeu de données BioCASe dans le logiciel et connectez-le à votre base de données. Configuez alors le jeu de données comme décrit dans le tutoriel [[DatasourceSetup]]. # Suiviez le tutoriel [[ABCD2Mapping pour créer un mapping]] pour le schéma de données (par exemple DarwinCore ou ABCD) que vous voulez utiliser. # Testez et débuggez votre web service BioCASe. Lisez le tutoriel [[Debugging]] pour plus d'informations à ce sujet. # Quand vous avez terminé avec l'installation et la configuration du BioCASE Provider Software, vous pouvez enregistrer le wen service BioCASe auprès du réseau sur lequel vous souhaitez publiser vos données, par exemple le [http://www.gbif.org GBIF]. Nous vous recommandons de contacter l'[[Special:Contact|équipe BioCASe]] pour vérifier au préalable votre web service nous pourrons alors diagnistiquer des problèmes plus compliqué à résoudre après cet enregistrement. == Support == En cas de difficultés lors de l'installation et de la configuration, sous devriez: * jeter un oeil sur le [[FAQ]] qui contient peut-être une réponse à votre question * demander l'aide au staff technique d'un projet lié à BioCASe (comme CABIN ou OpenUp!) * demander l'aide d'autres utilisateurs de BioCASe * contacter l'[[Special:Contact|BioCASe team]], qui est plutôt anglophone. Décrivez le plus précisément possible votre environnement (système d'exploitation et système de bases de données utlisés), l'étape où le problème apparaît, et tous les messages d'erreurs ou informations techniques retourné par le logiciel. Il est recommendé d'envoyer des copies d'écran à vos interlocuteurs pour ce type de problème. 23e51149372841470758a8fc65dbe7118254f4c7 508 507 2012-04-10T15:31:05Z FranckTheeten 15 /* Etapes */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' == Qu'est ce que le Logiciel Fournisseur de données BioCASe? == Le logiciel fournisseur de données BioCASe (abrégé '''BPS'' pour '''BioCASe Provider Software''' en anglais ) est un middleware, c'est à dire un logiciel situéà une position intermédiaire entre une base de données et une interface graphique. Il produit des données en XML, afin de les publier sur un réseau d'information scientifique. Après l'installation de BioCASe et sa configuration pour un type de base de données particulier, l'information publiée sera accessible sous la forme d'un web service de type BioCASe. Cela veut dire que l'information pourra être interrogée et recherchée avec des requêtes au format BioCASe, car BioCASe désigne également un protocole XML pour envoyer des messages à un web service. L'intérêt de ce type de logiciel est qu'il est "agnostique" par rapport au modèle de données de la base de données d'origine, c'est à dire que les requêtes envoyées au web services n'ont pas besoin de connaître la structure de la base de données pour fonctionner. Il peut être également utilisé avec n'importe quel type de schéma conceptuel de données en XML. Le BPS est un ensemble d'outils qui doivent être installée sur un serveur Web installé chez le fournisseur de données. Son composant central est le '''PyWrapper''', une interface XML/CGI, écrite en Python, pour se connecter à des bases de données . Il permet un accès standardisé (c'est à dire via des procédures uniformes qui peuvent être ainsi partagée par tous les acteurs d'un réseau d'information) à une large gamme de systèmes de base de données dont la structure est arbitraire (c'est à dire définie par le fournisseur de données seul). Un certain nombre d'outils est réuni autour du BPS pour configurer, tester et débugger une installation ou un web service BioCASe. Même si BioCASe peut être utilisé avec n'importe quel schéma conceptuel défini en XML, son domaine de prédilection est la publication de données d'occurrence provenant de bases de données contenant des spécimens ou des observations sur des réseaux d'information primaire sur la biodiversité, comme le [http://www.biocase.org réseau BioCASe] et le [http://www.gbif.org GBIF (Global Biodiversity Information Facility)]. == Prérequis pour utiliser le BPS == Avant d'utiliser le logiciel fournisseur de données BioCASe, trois conditions doivent être vérifiées: * D'une part, les données à publier doivent être stockées dans une système gestionnaires de base de données relationnelles (parfois abrégé '''SGBDR''' ou '''RDBMS''' en anglais) compatible avec le langage '''SQL'''. Son modèle de données doit être bien documenté, ou, au minimum, présent à l'esprit d'un gestionnaire de données. Les bases de données non compatibles avec SQL ne sont pas supportées par BioCASe. Actuellement, le logiciel peut se connecter au bases de données suivantes: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * Pour installer BioCASe, vous devez avoir un logiciel de serveur Web (Apache ou Microsoft IIS) accessible en permanence depuis l'Internet. Cela peut être le serveur d'une institution académique ou scientifique, ou un service d'hébergement. Il est parfois pénible dans ce second cas d'obtenir les droits d'installation auprès d'un service d'hébergement. Une connexion commutée peer to peer ou 'Dial-up' ne fonctionnera pas. Notez qu'il est tout à fait possible d'installer un serveur Internet sur votre propre machine, isolé du reste de l'Internet, et de faire fonctionner BioCASe dessus à des fins d'entraînement, de formation ou de développement, . Il sera alors accessible localement pour l'ordinateur seul (via le nom http://localhost ou l'adresse IP réservée 127.0.0.1) * Vous devez avoir la possibilité d'installer Python sur le serveur web en question (si jamais ce n'est pas le cas) ainsi que des librairies ou paquets complémentaires Python. Cela peut être effectué relativement facilement par un informaticien ou quelqu'un ayant de bonnes connaissances techniques, mais assurez vous que cela ne pose pas problèmes par rapport aux procédures et règles techniques qui pourraient être en vigueur dans votre institution. == Etapes == Une fois que vous vous êtes assurez que le BioCASe Provider Software est un choix pertinent pour vos besoins ([[Special:Contact|contactez l'équipe BioCASe]] si vous souhaitez plus d'information avant de prendre une décision), vous devez exécuter les étapes suivantes: # Pensez à l'information que vous souhaitez publier sur le réseau. Cela implique d'avoir réfléchi et répondu à la question de la propriété des données et des termes d'utilisation, licences, copyright et restrictions dans l'utilisation qui s'appliquent à leur publication. Vous devriez aussi vous demander si certaines information ne devraient '''pas''' être publiées car cela pourrait avoir une conséquence sur des espèces menacées. une solution possible est de soustraire certaines occurrence du jeu de données publier, ou de rendre moins précise une information exacte sur la localité d'observation. # Préparez la base de données comme décrit dans le tutoriel [[Preparation]]. C'est à cette étape que doit être prise la décision de publier soit directement la base de données principale qui est utilisée pour les mises à jour, soit un 'snapshot' (c'est à dire une copie de tout ou une partie des données prise à un moment précis), de créer une table de métadonnées (informations sur la base de données elle-même) et, dans certains cas, de transformer la présentation des données. # Suivez le tutoriel d'[[Installation]] pour installer le Fournisseur de données BioCASe. # Configurez un jeu de données BioCASe dans le logiciel et connectez-le à votre base de données. Configuez alors le jeu de données comme décrit dans le tutoriel [[DatasourceSetup]]. # Suiviez le tutoriel [[ABCD2Mapping pour créer un mapping]] pour le schéma de données (par exemple DarwinCore ou ABCD) que vous voulez utiliser. # Testez et débuggez votre web service BioCASe. Lisez le tutoriel [[Debugging]] pour plus d'informations à ce sujet. # Quand vous avez terminé avec l'installation et la configuration du BioCASE Provider Software, vous pouvez enregistrer le wen service BioCASe auprès du réseau sur lequel vous souhaitez publiser vos données, par exemple le [http://www.gbif.org GBIF]. Nous vous recommandons de contacter l'[[Special:Contact|équipe BioCASe]] pour vérifier au préalable votre web service nous pourrons alors diagnistiquer des problèmes plus compliqué à résoudre après cet enregistrement. == Support == En cas de difficultés lors de l'installation et de la configuration, sous devriez: * jeter un oeil sur le [[FAQ]] qui contient peut-être une réponse à votre question * demander l'aide au staff technique d'un projet lié à BioCASe (comme CABIN ou OpenUp!) * demander l'aide d'autres utilisateurs de BioCASe * contacter l'[[Special:Contact|BioCASe team]], qui est plutôt anglophone. Décrivez le plus précisément possible votre environnement (système d'exploitation et système de bases de données utlisés), l'étape où le problème apparaît, et tous les messages d'erreurs ou informations techniques retourné par le logiciel. Il est recommendé d'envoyer des copies d'écran à vos interlocuteurs pour ce type de problème. 8ebf9b55e6689d69c9f560b3d5aea3907a85f132 509 508 2012-04-10T15:31:42Z FranckTheeten 15 /* Etapes */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' == Qu'est ce que le Logiciel Fournisseur de données BioCASe? == Le logiciel fournisseur de données BioCASe (abrégé '''BPS'' pour '''BioCASe Provider Software''' en anglais ) est un middleware, c'est à dire un logiciel situéà une position intermédiaire entre une base de données et une interface graphique. Il produit des données en XML, afin de les publier sur un réseau d'information scientifique. Après l'installation de BioCASe et sa configuration pour un type de base de données particulier, l'information publiée sera accessible sous la forme d'un web service de type BioCASe. Cela veut dire que l'information pourra être interrogée et recherchée avec des requêtes au format BioCASe, car BioCASe désigne également un protocole XML pour envoyer des messages à un web service. L'intérêt de ce type de logiciel est qu'il est "agnostique" par rapport au modèle de données de la base de données d'origine, c'est à dire que les requêtes envoyées au web services n'ont pas besoin de connaître la structure de la base de données pour fonctionner. Il peut être également utilisé avec n'importe quel type de schéma conceptuel de données en XML. Le BPS est un ensemble d'outils qui doivent être installée sur un serveur Web installé chez le fournisseur de données. Son composant central est le '''PyWrapper''', une interface XML/CGI, écrite en Python, pour se connecter à des bases de données . Il permet un accès standardisé (c'est à dire via des procédures uniformes qui peuvent être ainsi partagée par tous les acteurs d'un réseau d'information) à une large gamme de systèmes de base de données dont la structure est arbitraire (c'est à dire définie par le fournisseur de données seul). Un certain nombre d'outils est réuni autour du BPS pour configurer, tester et débugger une installation ou un web service BioCASe. Même si BioCASe peut être utilisé avec n'importe quel schéma conceptuel défini en XML, son domaine de prédilection est la publication de données d'occurrence provenant de bases de données contenant des spécimens ou des observations sur des réseaux d'information primaire sur la biodiversité, comme le [http://www.biocase.org réseau BioCASe] et le [http://www.gbif.org GBIF (Global Biodiversity Information Facility)]. == Prérequis pour utiliser le BPS == Avant d'utiliser le logiciel fournisseur de données BioCASe, trois conditions doivent être vérifiées: * D'une part, les données à publier doivent être stockées dans une système gestionnaires de base de données relationnelles (parfois abrégé '''SGBDR''' ou '''RDBMS''' en anglais) compatible avec le langage '''SQL'''. Son modèle de données doit être bien documenté, ou, au minimum, présent à l'esprit d'un gestionnaire de données. Les bases de données non compatibles avec SQL ne sont pas supportées par BioCASe. Actuellement, le logiciel peut se connecter au bases de données suivantes: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * Pour installer BioCASe, vous devez avoir un logiciel de serveur Web (Apache ou Microsoft IIS) accessible en permanence depuis l'Internet. Cela peut être le serveur d'une institution académique ou scientifique, ou un service d'hébergement. Il est parfois pénible dans ce second cas d'obtenir les droits d'installation auprès d'un service d'hébergement. Une connexion commutée peer to peer ou 'Dial-up' ne fonctionnera pas. Notez qu'il est tout à fait possible d'installer un serveur Internet sur votre propre machine, isolé du reste de l'Internet, et de faire fonctionner BioCASe dessus à des fins d'entraînement, de formation ou de développement, . Il sera alors accessible localement pour l'ordinateur seul (via le nom http://localhost ou l'adresse IP réservée 127.0.0.1) * Vous devez avoir la possibilité d'installer Python sur le serveur web en question (si jamais ce n'est pas le cas) ainsi que des librairies ou paquets complémentaires Python. Cela peut être effectué relativement facilement par un informaticien ou quelqu'un ayant de bonnes connaissances techniques, mais assurez vous que cela ne pose pas problèmes par rapport aux procédures et règles techniques qui pourraient être en vigueur dans votre institution. == Etapes == Une fois que vous vous êtes assurez que le BioCASe Provider Software est un choix pertinent pour vos besoins ([[Special:Contact|contactez l'équipe BioCASe]] si vous souhaitez plus d'information avant de prendre une décision), vous devez exécuter les étapes suivantes: # Pensez à l'information que vous souhaitez publier sur le réseau. Cela implique d'avoir réfléchi et répondu à la question de la propriété des données et des termes d'utilisation, licences, copyright et restrictions dans l'utilisation qui s'appliquent à leur publication. Vous devriez aussi vous demander si certaines information ne devraient '''pas''' être publiées car cela pourrait avoir une conséquence sur des espèces menacées. une solution possible est de soustraire certaines occurrence du jeu de données publier, ou de rendre moins précise une information exacte sur la localité d'observation. # Préparez la base de données comme décrit dans le tutoriel [[Preparation]]. C'est à cette étape que doit être prise la décision de publier soit directement la base de données principale qui est utilisée pour les mises à jour, soit un 'snapshot' (c'est à dire une copie de tout ou une partie des données prise à un moment précis), de créer une table de métadonnées (informations sur la base de données elle-même) et, dans certains cas, de transformer la présentation des données. # Suivez le tutoriel d'[[Installation_(French) Installation]] pour installer le Fournisseur de données BioCASe. # Configurez un jeu de données BioCASe dans le logiciel et connectez-le à votre base de données. Configuez alors le jeu de données comme décrit dans le tutoriel [[DatasourceSetup]]. # Suiviez le tutoriel [[ABCD2Mapping pour créer un mapping]] pour le schéma de données (par exemple DarwinCore ou ABCD) que vous voulez utiliser. # Testez et débuggez votre web service BioCASe. Lisez le tutoriel [[Debugging]] pour plus d'informations à ce sujet. # Quand vous avez terminé avec l'installation et la configuration du BioCASE Provider Software, vous pouvez enregistrer le wen service BioCASe auprès du réseau sur lequel vous souhaitez publiser vos données, par exemple le [http://www.gbif.org GBIF]. Nous vous recommandons de contacter l'[[Special:Contact|équipe BioCASe]] pour vérifier au préalable votre web service nous pourrons alors diagnistiquer des problèmes plus compliqué à résoudre après cet enregistrement. == Support == En cas de difficultés lors de l'installation et de la configuration, sous devriez: * jeter un oeil sur le [[FAQ]] qui contient peut-être une réponse à votre question * demander l'aide au staff technique d'un projet lié à BioCASe (comme CABIN ou OpenUp!) * demander l'aide d'autres utilisateurs de BioCASe * contacter l'[[Special:Contact|BioCASe team]], qui est plutôt anglophone. Décrivez le plus précisément possible votre environnement (système d'exploitation et système de bases de données utlisés), l'étape où le problème apparaît, et tous les messages d'erreurs ou informations techniques retourné par le logiciel. Il est recommendé d'envoyer des copies d'écran à vos interlocuteurs pour ce type de problème. 0a5c64e25dfb5f1f2693f0d88e3e36d1b15d7d43 510 509 2012-04-10T15:32:14Z FranckTheeten 15 /* Etapes */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' == Qu'est ce que le Logiciel Fournisseur de données BioCASe? == Le logiciel fournisseur de données BioCASe (abrégé '''BPS'' pour '''BioCASe Provider Software''' en anglais ) est un middleware, c'est à dire un logiciel situéà une position intermédiaire entre une base de données et une interface graphique. Il produit des données en XML, afin de les publier sur un réseau d'information scientifique. Après l'installation de BioCASe et sa configuration pour un type de base de données particulier, l'information publiée sera accessible sous la forme d'un web service de type BioCASe. Cela veut dire que l'information pourra être interrogée et recherchée avec des requêtes au format BioCASe, car BioCASe désigne également un protocole XML pour envoyer des messages à un web service. L'intérêt de ce type de logiciel est qu'il est "agnostique" par rapport au modèle de données de la base de données d'origine, c'est à dire que les requêtes envoyées au web services n'ont pas besoin de connaître la structure de la base de données pour fonctionner. Il peut être également utilisé avec n'importe quel type de schéma conceptuel de données en XML. Le BPS est un ensemble d'outils qui doivent être installée sur un serveur Web installé chez le fournisseur de données. Son composant central est le '''PyWrapper''', une interface XML/CGI, écrite en Python, pour se connecter à des bases de données . Il permet un accès standardisé (c'est à dire via des procédures uniformes qui peuvent être ainsi partagée par tous les acteurs d'un réseau d'information) à une large gamme de systèmes de base de données dont la structure est arbitraire (c'est à dire définie par le fournisseur de données seul). Un certain nombre d'outils est réuni autour du BPS pour configurer, tester et débugger une installation ou un web service BioCASe. Même si BioCASe peut être utilisé avec n'importe quel schéma conceptuel défini en XML, son domaine de prédilection est la publication de données d'occurrence provenant de bases de données contenant des spécimens ou des observations sur des réseaux d'information primaire sur la biodiversité, comme le [http://www.biocase.org réseau BioCASe] et le [http://www.gbif.org GBIF (Global Biodiversity Information Facility)]. == Prérequis pour utiliser le BPS == Avant d'utiliser le logiciel fournisseur de données BioCASe, trois conditions doivent être vérifiées: * D'une part, les données à publier doivent être stockées dans une système gestionnaires de base de données relationnelles (parfois abrégé '''SGBDR''' ou '''RDBMS''' en anglais) compatible avec le langage '''SQL'''. Son modèle de données doit être bien documenté, ou, au minimum, présent à l'esprit d'un gestionnaire de données. Les bases de données non compatibles avec SQL ne sont pas supportées par BioCASe. Actuellement, le logiciel peut se connecter au bases de données suivantes: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * Pour installer BioCASe, vous devez avoir un logiciel de serveur Web (Apache ou Microsoft IIS) accessible en permanence depuis l'Internet. Cela peut être le serveur d'une institution académique ou scientifique, ou un service d'hébergement. Il est parfois pénible dans ce second cas d'obtenir les droits d'installation auprès d'un service d'hébergement. Une connexion commutée peer to peer ou 'Dial-up' ne fonctionnera pas. Notez qu'il est tout à fait possible d'installer un serveur Internet sur votre propre machine, isolé du reste de l'Internet, et de faire fonctionner BioCASe dessus à des fins d'entraînement, de formation ou de développement, . Il sera alors accessible localement pour l'ordinateur seul (via le nom http://localhost ou l'adresse IP réservée 127.0.0.1) * Vous devez avoir la possibilité d'installer Python sur le serveur web en question (si jamais ce n'est pas le cas) ainsi que des librairies ou paquets complémentaires Python. Cela peut être effectué relativement facilement par un informaticien ou quelqu'un ayant de bonnes connaissances techniques, mais assurez vous que cela ne pose pas problèmes par rapport aux procédures et règles techniques qui pourraient être en vigueur dans votre institution. == Etapes == Une fois que vous vous êtes assurez que le BioCASe Provider Software est un choix pertinent pour vos besoins ([[Special:Contact|contactez l'équipe BioCASe]] si vous souhaitez plus d'information avant de prendre une décision), vous devez exécuter les étapes suivantes: # Pensez à l'information que vous souhaitez publier sur le réseau. Cela implique d'avoir réfléchi et répondu à la question de la propriété des données et des termes d'utilisation, licences, copyright et restrictions dans l'utilisation qui s'appliquent à leur publication. Vous devriez aussi vous demander si certaines information ne devraient '''pas''' être publiées car cela pourrait avoir une conséquence sur des espèces menacées. une solution possible est de soustraire certaines occurrence du jeu de données publier, ou de rendre moins précise une information exacte sur la localité d'observation. # Préparez la base de données comme décrit dans le tutoriel [[Preparation]]. C'est à cette étape que doit être prise la décision de publier soit directement la base de données principale qui est utilisée pour les mises à jour, soit un 'snapshot' (c'est à dire une copie de tout ou une partie des données prise à un moment précis), de créer une table de métadonnées (informations sur la base de données elle-même) et, dans certains cas, de transformer la présentation des données. # Suivez le tutoriel d'[[Installation_(French)|Installation]] pour installer le Fournisseur de données BioCASe. # Configurez un jeu de données BioCASe dans le logiciel et connectez-le à votre base de données. Configuez alors le jeu de données comme décrit dans le tutoriel [[DatasourceSetup]]. # Suiviez le tutoriel [[ABCD2Mapping pour créer un mapping]] pour le schéma de données (par exemple DarwinCore ou ABCD) que vous voulez utiliser. # Testez et débuggez votre web service BioCASe. Lisez le tutoriel [[Debugging]] pour plus d'informations à ce sujet. # Quand vous avez terminé avec l'installation et la configuration du BioCASE Provider Software, vous pouvez enregistrer le wen service BioCASe auprès du réseau sur lequel vous souhaitez publiser vos données, par exemple le [http://www.gbif.org GBIF]. Nous vous recommandons de contacter l'[[Special:Contact|équipe BioCASe]] pour vérifier au préalable votre web service nous pourrons alors diagnistiquer des problèmes plus compliqué à résoudre après cet enregistrement. == Support == En cas de difficultés lors de l'installation et de la configuration, sous devriez: * jeter un oeil sur le [[FAQ]] qui contient peut-être une réponse à votre question * demander l'aide au staff technique d'un projet lié à BioCASe (comme CABIN ou OpenUp!) * demander l'aide d'autres utilisateurs de BioCASe * contacter l'[[Special:Contact|BioCASe team]], qui est plutôt anglophone. Décrivez le plus précisément possible votre environnement (système d'exploitation et système de bases de données utlisés), l'étape où le problème apparaît, et tous les messages d'erreurs ou informations techniques retourné par le logiciel. Il est recommendé d'envoyer des copies d'écran à vos interlocuteurs pour ce type de problème. dc163bdfc03c8c82b89136456128a787f4a7b1b4 511 510 2012-04-10T15:32:42Z FranckTheeten 15 /* Etapes */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' == Qu'est ce que le Logiciel Fournisseur de données BioCASe? == Le logiciel fournisseur de données BioCASe (abrégé '''BPS'' pour '''BioCASe Provider Software''' en anglais ) est un middleware, c'est à dire un logiciel situéà une position intermédiaire entre une base de données et une interface graphique. Il produit des données en XML, afin de les publier sur un réseau d'information scientifique. Après l'installation de BioCASe et sa configuration pour un type de base de données particulier, l'information publiée sera accessible sous la forme d'un web service de type BioCASe. Cela veut dire que l'information pourra être interrogée et recherchée avec des requêtes au format BioCASe, car BioCASe désigne également un protocole XML pour envoyer des messages à un web service. L'intérêt de ce type de logiciel est qu'il est "agnostique" par rapport au modèle de données de la base de données d'origine, c'est à dire que les requêtes envoyées au web services n'ont pas besoin de connaître la structure de la base de données pour fonctionner. Il peut être également utilisé avec n'importe quel type de schéma conceptuel de données en XML. Le BPS est un ensemble d'outils qui doivent être installée sur un serveur Web installé chez le fournisseur de données. Son composant central est le '''PyWrapper''', une interface XML/CGI, écrite en Python, pour se connecter à des bases de données . Il permet un accès standardisé (c'est à dire via des procédures uniformes qui peuvent être ainsi partagée par tous les acteurs d'un réseau d'information) à une large gamme de systèmes de base de données dont la structure est arbitraire (c'est à dire définie par le fournisseur de données seul). Un certain nombre d'outils est réuni autour du BPS pour configurer, tester et débugger une installation ou un web service BioCASe. Même si BioCASe peut être utilisé avec n'importe quel schéma conceptuel défini en XML, son domaine de prédilection est la publication de données d'occurrence provenant de bases de données contenant des spécimens ou des observations sur des réseaux d'information primaire sur la biodiversité, comme le [http://www.biocase.org réseau BioCASe] et le [http://www.gbif.org GBIF (Global Biodiversity Information Facility)]. == Prérequis pour utiliser le BPS == Avant d'utiliser le logiciel fournisseur de données BioCASe, trois conditions doivent être vérifiées: * D'une part, les données à publier doivent être stockées dans une système gestionnaires de base de données relationnelles (parfois abrégé '''SGBDR''' ou '''RDBMS''' en anglais) compatible avec le langage '''SQL'''. Son modèle de données doit être bien documenté, ou, au minimum, présent à l'esprit d'un gestionnaire de données. Les bases de données non compatibles avec SQL ne sont pas supportées par BioCASe. Actuellement, le logiciel peut se connecter au bases de données suivantes: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * Pour installer BioCASe, vous devez avoir un logiciel de serveur Web (Apache ou Microsoft IIS) accessible en permanence depuis l'Internet. Cela peut être le serveur d'une institution académique ou scientifique, ou un service d'hébergement. Il est parfois pénible dans ce second cas d'obtenir les droits d'installation auprès d'un service d'hébergement. Une connexion commutée peer to peer ou 'Dial-up' ne fonctionnera pas. Notez qu'il est tout à fait possible d'installer un serveur Internet sur votre propre machine, isolé du reste de l'Internet, et de faire fonctionner BioCASe dessus à des fins d'entraînement, de formation ou de développement, . Il sera alors accessible localement pour l'ordinateur seul (via le nom http://localhost ou l'adresse IP réservée 127.0.0.1) * Vous devez avoir la possibilité d'installer Python sur le serveur web en question (si jamais ce n'est pas le cas) ainsi que des librairies ou paquets complémentaires Python. Cela peut être effectué relativement facilement par un informaticien ou quelqu'un ayant de bonnes connaissances techniques, mais assurez vous que cela ne pose pas problèmes par rapport aux procédures et règles techniques qui pourraient être en vigueur dans votre institution. == Etapes == Une fois que vous vous êtes assurez que le BioCASe Provider Software est un choix pertinent pour vos besoins ([[Special:Contact|contactez l'équipe BioCASe]] si vous souhaitez plus d'information avant de prendre une décision), vous devez exécuter les étapes suivantes: # Pensez à l'information que vous souhaitez publier sur le réseau. Cela implique d'avoir réfléchi et répondu à la question de la propriété des données et des termes d'utilisation, licences, copyright et restrictions dans l'utilisation qui s'appliquent à leur publication. Vous devriez aussi vous demander si certaines information ne devraient '''pas''' être publiées car cela pourrait avoir une conséquence sur des espèces menacées. une solution possible est de soustraire certaines occurrence du jeu de données publier, ou de rendre moins précise une information exacte sur la localité d'observation. # Préparez la base de données comme décrit dans le tutoriel [[Preparation]]. C'est à cette étape que doit être prise la décision de publier soit directement la base de données principale qui est utilisée pour les mises à jour, soit un 'snapshot' (c'est à dire une copie de tout ou une partie des données prise à un moment précis), de créer une table de métadonnées (informations sur la base de données elle-même) et, dans certains cas, de transformer la présentation des données. # Suivez le tutoriel d'[[Installation_(French)|Installation]] pour installer le Fournisseur de données BioCASe. # Configurez un jeu de données BioCASe viale logiciel et connectez-le à votre base de données. Configurez alors le jeu de données comme décrit dans le tutoriel [[DatasourceSetup]]. # Suiviez le tutoriel [[ABCD2Mapping pour créer un mapping]] pour le schéma de données (par exemple DarwinCore ou ABCD) que vous voulez utiliser. # Testez et débuggez votre web service BioCASe. Lisez le tutoriel [[Debugging]] pour plus d'informations à ce sujet. # Quand vous avez terminé avec l'installation et la configuration du BioCASE Provider Software, vous pouvez enregistrer le wen service BioCASe auprès du réseau sur lequel vous souhaitez publiser vos données, par exemple le [http://www.gbif.org GBIF]. Nous vous recommandons de contacter l'[[Special:Contact|équipe BioCASe]] pour vérifier au préalable votre web service nous pourrons alors diagnistiquer des problèmes plus compliqué à résoudre après cet enregistrement. == Support == En cas de difficultés lors de l'installation et de la configuration, sous devriez: * jeter un oeil sur le [[FAQ]] qui contient peut-être une réponse à votre question * demander l'aide au staff technique d'un projet lié à BioCASe (comme CABIN ou OpenUp!) * demander l'aide d'autres utilisateurs de BioCASe * contacter l'[[Special:Contact|BioCASe team]], qui est plutôt anglophone. Décrivez le plus précisément possible votre environnement (système d'exploitation et système de bases de données utlisés), l'étape où le problème apparaît, et tous les messages d'erreurs ou informations techniques retourné par le logiciel. Il est recommendé d'envoyer des copies d'écran à vos interlocuteurs pour ce type de problème. 01f399dda50948a6a132ed256396b62493a638a3 512 511 2012-04-10T15:33:50Z FranckTheeten 15 /* Etapes */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' == Qu'est ce que le Logiciel Fournisseur de données BioCASe? == Le logiciel fournisseur de données BioCASe (abrégé '''BPS'' pour '''BioCASe Provider Software''' en anglais ) est un middleware, c'est à dire un logiciel situéà une position intermédiaire entre une base de données et une interface graphique. Il produit des données en XML, afin de les publier sur un réseau d'information scientifique. Après l'installation de BioCASe et sa configuration pour un type de base de données particulier, l'information publiée sera accessible sous la forme d'un web service de type BioCASe. Cela veut dire que l'information pourra être interrogée et recherchée avec des requêtes au format BioCASe, car BioCASe désigne également un protocole XML pour envoyer des messages à un web service. L'intérêt de ce type de logiciel est qu'il est "agnostique" par rapport au modèle de données de la base de données d'origine, c'est à dire que les requêtes envoyées au web services n'ont pas besoin de connaître la structure de la base de données pour fonctionner. Il peut être également utilisé avec n'importe quel type de schéma conceptuel de données en XML. Le BPS est un ensemble d'outils qui doivent être installée sur un serveur Web installé chez le fournisseur de données. Son composant central est le '''PyWrapper''', une interface XML/CGI, écrite en Python, pour se connecter à des bases de données . Il permet un accès standardisé (c'est à dire via des procédures uniformes qui peuvent être ainsi partagée par tous les acteurs d'un réseau d'information) à une large gamme de systèmes de base de données dont la structure est arbitraire (c'est à dire définie par le fournisseur de données seul). Un certain nombre d'outils est réuni autour du BPS pour configurer, tester et débugger une installation ou un web service BioCASe. Même si BioCASe peut être utilisé avec n'importe quel schéma conceptuel défini en XML, son domaine de prédilection est la publication de données d'occurrence provenant de bases de données contenant des spécimens ou des observations sur des réseaux d'information primaire sur la biodiversité, comme le [http://www.biocase.org réseau BioCASe] et le [http://www.gbif.org GBIF (Global Biodiversity Information Facility)]. == Prérequis pour utiliser le BPS == Avant d'utiliser le logiciel fournisseur de données BioCASe, trois conditions doivent être vérifiées: * D'une part, les données à publier doivent être stockées dans une système gestionnaires de base de données relationnelles (parfois abrégé '''SGBDR''' ou '''RDBMS''' en anglais) compatible avec le langage '''SQL'''. Son modèle de données doit être bien documenté, ou, au minimum, présent à l'esprit d'un gestionnaire de données. Les bases de données non compatibles avec SQL ne sont pas supportées par BioCASe. Actuellement, le logiciel peut se connecter au bases de données suivantes: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * Pour installer BioCASe, vous devez avoir un logiciel de serveur Web (Apache ou Microsoft IIS) accessible en permanence depuis l'Internet. Cela peut être le serveur d'une institution académique ou scientifique, ou un service d'hébergement. Il est parfois pénible dans ce second cas d'obtenir les droits d'installation auprès d'un service d'hébergement. Une connexion commutée peer to peer ou 'Dial-up' ne fonctionnera pas. Notez qu'il est tout à fait possible d'installer un serveur Internet sur votre propre machine, isolé du reste de l'Internet, et de faire fonctionner BioCASe dessus à des fins d'entraînement, de formation ou de développement, . Il sera alors accessible localement pour l'ordinateur seul (via le nom http://localhost ou l'adresse IP réservée 127.0.0.1) * Vous devez avoir la possibilité d'installer Python sur le serveur web en question (si jamais ce n'est pas le cas) ainsi que des librairies ou paquets complémentaires Python. Cela peut être effectué relativement facilement par un informaticien ou quelqu'un ayant de bonnes connaissances techniques, mais assurez vous que cela ne pose pas problèmes par rapport aux procédures et règles techniques qui pourraient être en vigueur dans votre institution. == Etapes == Une fois que vous vous êtes assurez que le BioCASe Provider Software est un choix pertinent pour vos besoins ([[Special:Contact|contactez l'équipe BioCASe]] si vous souhaitez plus d'information avant de prendre une décision), vous devez exécuter les étapes suivantes: # Pensez à l'information que vous souhaitez publier sur le réseau. Cela implique d'avoir réfléchi et répondu à la question de la propriété des données et des termes d'utilisation, licences, copyright et restrictions dans l'utilisation qui s'appliquent à leur publication. Vous devriez aussi vous demander si certaines information ne devraient '''pas''' être publiées car cela pourrait avoir une conséquence sur des espèces menacées. une solution possible est de soustraire certaines occurrence du jeu de données publier, ou de rendre moins précise une information exacte sur la localité d'observation. # Préparez la base de données comme décrit dans le tutoriel [[Preparation]]. C'est à cette étape que doit être prise la décision de publier soit directement la base de données principale qui est utilisée pour les mises à jour, soit un 'snapshot' (c'est à dire une copie de tout ou une partie des données prise à un moment précis), de créer une table de métadonnées (informations sur la base de données elle-même) et, dans certains cas, de transformer la présentation des données. # Suivez le tutoriel d'[[Installation_(French)|Installation]] pour installer le Fournisseur de données BioCASe. # Configurez un jeu de données BioCASe viale logiciel et connectez-le à votre base de données. Configurez alors le jeu de données comme décrit dans le tutoriel [[DatasourceSetup]]. # Suiviez le tutoriel [[ABCD2Mapping| ABCD2Mapping pour créer un mapping]] (un mapping est une correspondance entre la structure de votre bases de données et un schéma XML) pour le schéma de données (par exemple DarwinCore ou ABCD) que vous voulez utiliser. # Testez et débuggez votre web service BioCASe. Lisez le tutoriel [[Debugging]] pour plus d'informations à ce sujet. # Quand vous avez terminé avec l'installation et la configuration du BioCASE Provider Software, vous pouvez enregistrer le wen service BioCASe auprès du réseau sur lequel vous souhaitez publiser vos données, par exemple le [http://www.gbif.org GBIF]. Nous vous recommandons de contacter l'[[Special:Contact|équipe BioCASe]] pour vérifier au préalable votre web service nous pourrons alors diagnistiquer des problèmes plus compliqué à résoudre après cet enregistrement. == Support == En cas de difficultés lors de l'installation et de la configuration, sous devriez: * jeter un oeil sur le [[FAQ]] qui contient peut-être une réponse à votre question * demander l'aide au staff technique d'un projet lié à BioCASe (comme CABIN ou OpenUp!) * demander l'aide d'autres utilisateurs de BioCASe * contacter l'[[Special:Contact|BioCASe team]], qui est plutôt anglophone. Décrivez le plus précisément possible votre environnement (système d'exploitation et système de bases de données utlisés), l'étape où le problème apparaît, et tous les messages d'erreurs ou informations techniques retourné par le logiciel. Il est recommendé d'envoyer des copies d'écran à vos interlocuteurs pour ce type de problème. cc06906c0ea31f36aa3c717a427073c683e687c9 513 512 2012-04-10T15:34:32Z FranckTheeten 15 /* Etapes */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' == Qu'est ce que le Logiciel Fournisseur de données BioCASe? == Le logiciel fournisseur de données BioCASe (abrégé '''BPS'' pour '''BioCASe Provider Software''' en anglais ) est un middleware, c'est à dire un logiciel situéà une position intermédiaire entre une base de données et une interface graphique. Il produit des données en XML, afin de les publier sur un réseau d'information scientifique. Après l'installation de BioCASe et sa configuration pour un type de base de données particulier, l'information publiée sera accessible sous la forme d'un web service de type BioCASe. Cela veut dire que l'information pourra être interrogée et recherchée avec des requêtes au format BioCASe, car BioCASe désigne également un protocole XML pour envoyer des messages à un web service. L'intérêt de ce type de logiciel est qu'il est "agnostique" par rapport au modèle de données de la base de données d'origine, c'est à dire que les requêtes envoyées au web services n'ont pas besoin de connaître la structure de la base de données pour fonctionner. Il peut être également utilisé avec n'importe quel type de schéma conceptuel de données en XML. Le BPS est un ensemble d'outils qui doivent être installée sur un serveur Web installé chez le fournisseur de données. Son composant central est le '''PyWrapper''', une interface XML/CGI, écrite en Python, pour se connecter à des bases de données . Il permet un accès standardisé (c'est à dire via des procédures uniformes qui peuvent être ainsi partagée par tous les acteurs d'un réseau d'information) à une large gamme de systèmes de base de données dont la structure est arbitraire (c'est à dire définie par le fournisseur de données seul). Un certain nombre d'outils est réuni autour du BPS pour configurer, tester et débugger une installation ou un web service BioCASe. Même si BioCASe peut être utilisé avec n'importe quel schéma conceptuel défini en XML, son domaine de prédilection est la publication de données d'occurrence provenant de bases de données contenant des spécimens ou des observations sur des réseaux d'information primaire sur la biodiversité, comme le [http://www.biocase.org réseau BioCASe] et le [http://www.gbif.org GBIF (Global Biodiversity Information Facility)]. == Prérequis pour utiliser le BPS == Avant d'utiliser le logiciel fournisseur de données BioCASe, trois conditions doivent être vérifiées: * D'une part, les données à publier doivent être stockées dans une système gestionnaires de base de données relationnelles (parfois abrégé '''SGBDR''' ou '''RDBMS''' en anglais) compatible avec le langage '''SQL'''. Son modèle de données doit être bien documenté, ou, au minimum, présent à l'esprit d'un gestionnaire de données. Les bases de données non compatibles avec SQL ne sont pas supportées par BioCASe. Actuellement, le logiciel peut se connecter au bases de données suivantes: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * Pour installer BioCASe, vous devez avoir un logiciel de serveur Web (Apache ou Microsoft IIS) accessible en permanence depuis l'Internet. Cela peut être le serveur d'une institution académique ou scientifique, ou un service d'hébergement. Il est parfois pénible dans ce second cas d'obtenir les droits d'installation auprès d'un service d'hébergement. Une connexion commutée peer to peer ou 'Dial-up' ne fonctionnera pas. Notez qu'il est tout à fait possible d'installer un serveur Internet sur votre propre machine, isolé du reste de l'Internet, et de faire fonctionner BioCASe dessus à des fins d'entraînement, de formation ou de développement, . Il sera alors accessible localement pour l'ordinateur seul (via le nom http://localhost ou l'adresse IP réservée 127.0.0.1) * Vous devez avoir la possibilité d'installer Python sur le serveur web en question (si jamais ce n'est pas le cas) ainsi que des librairies ou paquets complémentaires Python. Cela peut être effectué relativement facilement par un informaticien ou quelqu'un ayant de bonnes connaissances techniques, mais assurez vous que cela ne pose pas problèmes par rapport aux procédures et règles techniques qui pourraient être en vigueur dans votre institution. == Etapes == Une fois que vous vous êtes assurez que le BioCASe Provider Software est un choix pertinent pour vos besoins ([[Special:Contact|contactez l'équipe BioCASe]] si vous souhaitez plus d'information avant de prendre une décision), vous devez exécuter les étapes suivantes: # Pensez à l'information que vous souhaitez publier sur le réseau. Cela implique d'avoir réfléchi et répondu à la question de la propriété des données et des termes d'utilisation, licences, copyright et restrictions dans l'utilisation qui s'appliquent à leur publication. Vous devriez aussi vous demander si certaines information ne devraient '''pas''' être publiées car cela pourrait avoir une conséquence sur des espèces menacées. une solution possible est de soustraire certaines occurrence du jeu de données publier, ou de rendre moins précise une information exacte sur la localité d'observation. # Préparez la base de données comme décrit dans le tutoriel [[Preparation]]. C'est à cette étape que doit être prise la décision de publier soit directement la base de données principale qui est utilisée pour les mises à jour, soit un 'snapshot' (c'est à dire une copie de tout ou une partie des données prise à un moment précis), de créer une table de métadonnées (informations sur la base de données elle-même) et, dans certains cas, de transformer la présentation des données. # Suivez le tutoriel d'[[Installation_(French)|Installation]] pour installer le Fournisseur de données BioCASe. # Configurez un jeu de données BioCASe viale logiciel et connectez-le à votre base de données. Configurez alors le jeu de données comme décrit dans le tutoriel [[DatasourceSetup]]. # Suiviez le tutoriel [[ABCD2Mapping| ABCD2Mapping (pur l'instant en anglais)]] pour créer un mapping (un mapping est une correspondance entre la structure de votre bases de données et un schéma XML) pour le schéma de données (par exemple DarwinCore ou ABCD) que vous voulez utiliser. # Testez et débuggez votre web service BioCASe. Lisez le tutoriel [[Debugging]] pour plus d'informations à ce sujet. # Quand vous avez terminé avec l'installation et la configuration du BioCASE Provider Software, vous pouvez enregistrer le wen service BioCASe auprès du réseau sur lequel vous souhaitez publiser vos données, par exemple le [http://www.gbif.org GBIF]. Nous vous recommandons de contacter l'[[Special:Contact|équipe BioCASe]] pour vérifier au préalable votre web service nous pourrons alors diagnistiquer des problèmes plus compliqué à résoudre après cet enregistrement. == Support == En cas de difficultés lors de l'installation et de la configuration, sous devriez: * jeter un oeil sur le [[FAQ]] qui contient peut-être une réponse à votre question * demander l'aide au staff technique d'un projet lié à BioCASe (comme CABIN ou OpenUp!) * demander l'aide d'autres utilisateurs de BioCASe * contacter l'[[Special:Contact|BioCASe team]], qui est plutôt anglophone. Décrivez le plus précisément possible votre environnement (système d'exploitation et système de bases de données utlisés), l'étape où le problème apparaît, et tous les messages d'erreurs ou informations techniques retourné par le logiciel. Il est recommendé d'envoyer des copies d'écran à vos interlocuteurs pour ce type de problème. a1efae05a71683654fbe3610b1e16abea67b5dac 514 513 2012-04-10T15:34:49Z FranckTheeten 15 /* Etapes */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' == Qu'est ce que le Logiciel Fournisseur de données BioCASe? == Le logiciel fournisseur de données BioCASe (abrégé '''BPS'' pour '''BioCASe Provider Software''' en anglais ) est un middleware, c'est à dire un logiciel situéà une position intermédiaire entre une base de données et une interface graphique. Il produit des données en XML, afin de les publier sur un réseau d'information scientifique. Après l'installation de BioCASe et sa configuration pour un type de base de données particulier, l'information publiée sera accessible sous la forme d'un web service de type BioCASe. Cela veut dire que l'information pourra être interrogée et recherchée avec des requêtes au format BioCASe, car BioCASe désigne également un protocole XML pour envoyer des messages à un web service. L'intérêt de ce type de logiciel est qu'il est "agnostique" par rapport au modèle de données de la base de données d'origine, c'est à dire que les requêtes envoyées au web services n'ont pas besoin de connaître la structure de la base de données pour fonctionner. Il peut être également utilisé avec n'importe quel type de schéma conceptuel de données en XML. Le BPS est un ensemble d'outils qui doivent être installée sur un serveur Web installé chez le fournisseur de données. Son composant central est le '''PyWrapper''', une interface XML/CGI, écrite en Python, pour se connecter à des bases de données . Il permet un accès standardisé (c'est à dire via des procédures uniformes qui peuvent être ainsi partagée par tous les acteurs d'un réseau d'information) à une large gamme de systèmes de base de données dont la structure est arbitraire (c'est à dire définie par le fournisseur de données seul). Un certain nombre d'outils est réuni autour du BPS pour configurer, tester et débugger une installation ou un web service BioCASe. Même si BioCASe peut être utilisé avec n'importe quel schéma conceptuel défini en XML, son domaine de prédilection est la publication de données d'occurrence provenant de bases de données contenant des spécimens ou des observations sur des réseaux d'information primaire sur la biodiversité, comme le [http://www.biocase.org réseau BioCASe] et le [http://www.gbif.org GBIF (Global Biodiversity Information Facility)]. == Prérequis pour utiliser le BPS == Avant d'utiliser le logiciel fournisseur de données BioCASe, trois conditions doivent être vérifiées: * D'une part, les données à publier doivent être stockées dans une système gestionnaires de base de données relationnelles (parfois abrégé '''SGBDR''' ou '''RDBMS''' en anglais) compatible avec le langage '''SQL'''. Son modèle de données doit être bien documenté, ou, au minimum, présent à l'esprit d'un gestionnaire de données. Les bases de données non compatibles avec SQL ne sont pas supportées par BioCASe. Actuellement, le logiciel peut se connecter au bases de données suivantes: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * Pour installer BioCASe, vous devez avoir un logiciel de serveur Web (Apache ou Microsoft IIS) accessible en permanence depuis l'Internet. Cela peut être le serveur d'une institution académique ou scientifique, ou un service d'hébergement. Il est parfois pénible dans ce second cas d'obtenir les droits d'installation auprès d'un service d'hébergement. Une connexion commutée peer to peer ou 'Dial-up' ne fonctionnera pas. Notez qu'il est tout à fait possible d'installer un serveur Internet sur votre propre machine, isolé du reste de l'Internet, et de faire fonctionner BioCASe dessus à des fins d'entraînement, de formation ou de développement, . Il sera alors accessible localement pour l'ordinateur seul (via le nom http://localhost ou l'adresse IP réservée 127.0.0.1) * Vous devez avoir la possibilité d'installer Python sur le serveur web en question (si jamais ce n'est pas le cas) ainsi que des librairies ou paquets complémentaires Python. Cela peut être effectué relativement facilement par un informaticien ou quelqu'un ayant de bonnes connaissances techniques, mais assurez vous que cela ne pose pas problèmes par rapport aux procédures et règles techniques qui pourraient être en vigueur dans votre institution. == Etapes == Une fois que vous vous êtes assurez que le BioCASe Provider Software est un choix pertinent pour vos besoins ([[Special:Contact|contactez l'équipe BioCASe]] si vous souhaitez plus d'information avant de prendre une décision), vous devez exécuter les étapes suivantes: # Pensez à l'information que vous souhaitez publier sur le réseau. Cela implique d'avoir réfléchi et répondu à la question de la propriété des données et des termes d'utilisation, licences, copyright et restrictions dans l'utilisation qui s'appliquent à leur publication. Vous devriez aussi vous demander si certaines information ne devraient '''pas''' être publiées car cela pourrait avoir une conséquence sur des espèces menacées. une solution possible est de soustraire certaines occurrence du jeu de données publier, ou de rendre moins précise une information exacte sur la localité d'observation. # Préparez la base de données comme décrit dans le tutoriel [[Preparation]]. C'est à cette étape que doit être prise la décision de publier soit directement la base de données principale qui est utilisée pour les mises à jour, soit un 'snapshot' (c'est à dire une copie de tout ou une partie des données prise à un moment précis), de créer une table de métadonnées (informations sur la base de données elle-même) et, dans certains cas, de transformer la présentation des données. # Suivez le tutoriel d'[[Installation_(French)|Installation]] pour installer le Fournisseur de données BioCASe. # Configurez un jeu de données BioCASe viale logiciel et connectez-le à votre base de données. Configurez alors le jeu de données comme décrit dans le tutoriel [[DatasourceSetup]]. # Suiviez le tutoriel [[ABCD2Mapping| ABCD2Mapping (pour l'instant en anglais)]] pour créer un mapping (un mapping est une correspondance entre la structure de votre bases de données et un schéma XML) pour le schéma de données (par exemple DarwinCore ou ABCD) que vous voulez utiliser. # Testez et débuggez votre web service BioCASe. Lisez le tutoriel [[Debugging]] pour plus d'informations à ce sujet. # Quand vous avez terminé avec l'installation et la configuration du BioCASE Provider Software, vous pouvez enregistrer le wen service BioCASe auprès du réseau sur lequel vous souhaitez publiser vos données, par exemple le [http://www.gbif.org GBIF]. Nous vous recommandons de contacter l'[[Special:Contact|équipe BioCASe]] pour vérifier au préalable votre web service nous pourrons alors diagnistiquer des problèmes plus compliqué à résoudre après cet enregistrement. == Support == En cas de difficultés lors de l'installation et de la configuration, sous devriez: * jeter un oeil sur le [[FAQ]] qui contient peut-être une réponse à votre question * demander l'aide au staff technique d'un projet lié à BioCASe (comme CABIN ou OpenUp!) * demander l'aide d'autres utilisateurs de BioCASe * contacter l'[[Special:Contact|BioCASe team]], qui est plutôt anglophone. Décrivez le plus précisément possible votre environnement (système d'exploitation et système de bases de données utlisés), l'étape où le problème apparaît, et tous les messages d'erreurs ou informations techniques retourné par le logiciel. Il est recommendé d'envoyer des copies d'écran à vos interlocuteurs pour ce type de problème. c67b503f4f93d90980f8a60067fc70a8dfdcc98e 515 514 2012-04-10T15:34:59Z FranckTheeten 15 /* Etapes */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' == Qu'est ce que le Logiciel Fournisseur de données BioCASe? == Le logiciel fournisseur de données BioCASe (abrégé '''BPS'' pour '''BioCASe Provider Software''' en anglais ) est un middleware, c'est à dire un logiciel situéà une position intermédiaire entre une base de données et une interface graphique. Il produit des données en XML, afin de les publier sur un réseau d'information scientifique. Après l'installation de BioCASe et sa configuration pour un type de base de données particulier, l'information publiée sera accessible sous la forme d'un web service de type BioCASe. Cela veut dire que l'information pourra être interrogée et recherchée avec des requêtes au format BioCASe, car BioCASe désigne également un protocole XML pour envoyer des messages à un web service. L'intérêt de ce type de logiciel est qu'il est "agnostique" par rapport au modèle de données de la base de données d'origine, c'est à dire que les requêtes envoyées au web services n'ont pas besoin de connaître la structure de la base de données pour fonctionner. Il peut être également utilisé avec n'importe quel type de schéma conceptuel de données en XML. Le BPS est un ensemble d'outils qui doivent être installée sur un serveur Web installé chez le fournisseur de données. Son composant central est le '''PyWrapper''', une interface XML/CGI, écrite en Python, pour se connecter à des bases de données . Il permet un accès standardisé (c'est à dire via des procédures uniformes qui peuvent être ainsi partagée par tous les acteurs d'un réseau d'information) à une large gamme de systèmes de base de données dont la structure est arbitraire (c'est à dire définie par le fournisseur de données seul). Un certain nombre d'outils est réuni autour du BPS pour configurer, tester et débugger une installation ou un web service BioCASe. Même si BioCASe peut être utilisé avec n'importe quel schéma conceptuel défini en XML, son domaine de prédilection est la publication de données d'occurrence provenant de bases de données contenant des spécimens ou des observations sur des réseaux d'information primaire sur la biodiversité, comme le [http://www.biocase.org réseau BioCASe] et le [http://www.gbif.org GBIF (Global Biodiversity Information Facility)]. == Prérequis pour utiliser le BPS == Avant d'utiliser le logiciel fournisseur de données BioCASe, trois conditions doivent être vérifiées: * D'une part, les données à publier doivent être stockées dans une système gestionnaires de base de données relationnelles (parfois abrégé '''SGBDR''' ou '''RDBMS''' en anglais) compatible avec le langage '''SQL'''. Son modèle de données doit être bien documenté, ou, au minimum, présent à l'esprit d'un gestionnaire de données. Les bases de données non compatibles avec SQL ne sont pas supportées par BioCASe. Actuellement, le logiciel peut se connecter au bases de données suivantes: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * Pour installer BioCASe, vous devez avoir un logiciel de serveur Web (Apache ou Microsoft IIS) accessible en permanence depuis l'Internet. Cela peut être le serveur d'une institution académique ou scientifique, ou un service d'hébergement. Il est parfois pénible dans ce second cas d'obtenir les droits d'installation auprès d'un service d'hébergement. Une connexion commutée peer to peer ou 'Dial-up' ne fonctionnera pas. Notez qu'il est tout à fait possible d'installer un serveur Internet sur votre propre machine, isolé du reste de l'Internet, et de faire fonctionner BioCASe dessus à des fins d'entraînement, de formation ou de développement, . Il sera alors accessible localement pour l'ordinateur seul (via le nom http://localhost ou l'adresse IP réservée 127.0.0.1) * Vous devez avoir la possibilité d'installer Python sur le serveur web en question (si jamais ce n'est pas le cas) ainsi que des librairies ou paquets complémentaires Python. Cela peut être effectué relativement facilement par un informaticien ou quelqu'un ayant de bonnes connaissances techniques, mais assurez vous que cela ne pose pas problèmes par rapport aux procédures et règles techniques qui pourraient être en vigueur dans votre institution. == Etapes == Une fois que vous vous êtes assurez que le BioCASe Provider Software est un choix pertinent pour vos besoins ([[Special:Contact|contactez l'équipe BioCASe]] si vous souhaitez plus d'information avant de prendre une décision), vous devez exécuter les étapes suivantes: # Pensez à l'information que vous souhaitez publier sur le réseau. Cela implique d'avoir réfléchi et répondu à la question de la propriété des données et des termes d'utilisation, licences, copyright et restrictions dans l'utilisation qui s'appliquent à leur publication. Vous devriez aussi vous demander si certaines information ne devraient '''pas''' être publiées car cela pourrait avoir une conséquence sur des espèces menacées. une solution possible est de soustraire certaines occurrence du jeu de données publier, ou de rendre moins précise une information exacte sur la localité d'observation. # Préparez la base de données comme décrit dans le tutoriel [[Preparation]]. C'est à cette étape que doit être prise la décision de publier soit directement la base de données principale qui est utilisée pour les mises à jour, soit un 'snapshot' (c'est à dire une copie de tout ou une partie des données prise à un moment précis), de créer une table de métadonnées (informations sur la base de données elle-même) et, dans certains cas, de transformer la présentation des données. # Suivez le tutoriel d'[[Installation_(French)|Installation]] pour installer le Fournisseur de données BioCASe. # Configurez un jeu de données BioCASe viale logiciel et connectez-le à votre base de données. Configurez alors le jeu de données comme décrit dans le tutoriel [[DatasourceSetup]]. # Suivez le tutoriel [[ABCD2Mapping| ABCD2Mapping (pour l'instant en anglais)]] pour créer un mapping (un mapping est une correspondance entre la structure de votre bases de données et un schéma XML) pour le schéma de données (par exemple DarwinCore ou ABCD) que vous voulez utiliser. # Testez et débuggez votre web service BioCASe. Lisez le tutoriel [[Debugging]] pour plus d'informations à ce sujet. # Quand vous avez terminé avec l'installation et la configuration du BioCASE Provider Software, vous pouvez enregistrer le wen service BioCASe auprès du réseau sur lequel vous souhaitez publiser vos données, par exemple le [http://www.gbif.org GBIF]. Nous vous recommandons de contacter l'[[Special:Contact|équipe BioCASe]] pour vérifier au préalable votre web service nous pourrons alors diagnistiquer des problèmes plus compliqué à résoudre après cet enregistrement. == Support == En cas de difficultés lors de l'installation et de la configuration, sous devriez: * jeter un oeil sur le [[FAQ]] qui contient peut-être une réponse à votre question * demander l'aide au staff technique d'un projet lié à BioCASe (comme CABIN ou OpenUp!) * demander l'aide d'autres utilisateurs de BioCASe * contacter l'[[Special:Contact|BioCASe team]], qui est plutôt anglophone. Décrivez le plus précisément possible votre environnement (système d'exploitation et système de bases de données utlisés), l'étape où le problème apparaît, et tous les messages d'erreurs ou informations techniques retourné par le logiciel. Il est recommendé d'envoyer des copies d'écran à vos interlocuteurs pour ce type de problème. 839df2bea1a0576a0a8130712d444341779fe93a 516 515 2012-04-10T15:36:22Z FranckTheeten 15 /* Etapes */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' == Qu'est ce que le Logiciel Fournisseur de données BioCASe? == Le logiciel fournisseur de données BioCASe (abrégé '''BPS'' pour '''BioCASe Provider Software''' en anglais ) est un middleware, c'est à dire un logiciel situéà une position intermédiaire entre une base de données et une interface graphique. Il produit des données en XML, afin de les publier sur un réseau d'information scientifique. Après l'installation de BioCASe et sa configuration pour un type de base de données particulier, l'information publiée sera accessible sous la forme d'un web service de type BioCASe. Cela veut dire que l'information pourra être interrogée et recherchée avec des requêtes au format BioCASe, car BioCASe désigne également un protocole XML pour envoyer des messages à un web service. L'intérêt de ce type de logiciel est qu'il est "agnostique" par rapport au modèle de données de la base de données d'origine, c'est à dire que les requêtes envoyées au web services n'ont pas besoin de connaître la structure de la base de données pour fonctionner. Il peut être également utilisé avec n'importe quel type de schéma conceptuel de données en XML. Le BPS est un ensemble d'outils qui doivent être installée sur un serveur Web installé chez le fournisseur de données. Son composant central est le '''PyWrapper''', une interface XML/CGI, écrite en Python, pour se connecter à des bases de données . Il permet un accès standardisé (c'est à dire via des procédures uniformes qui peuvent être ainsi partagée par tous les acteurs d'un réseau d'information) à une large gamme de systèmes de base de données dont la structure est arbitraire (c'est à dire définie par le fournisseur de données seul). Un certain nombre d'outils est réuni autour du BPS pour configurer, tester et débugger une installation ou un web service BioCASe. Même si BioCASe peut être utilisé avec n'importe quel schéma conceptuel défini en XML, son domaine de prédilection est la publication de données d'occurrence provenant de bases de données contenant des spécimens ou des observations sur des réseaux d'information primaire sur la biodiversité, comme le [http://www.biocase.org réseau BioCASe] et le [http://www.gbif.org GBIF (Global Biodiversity Information Facility)]. == Prérequis pour utiliser le BPS == Avant d'utiliser le logiciel fournisseur de données BioCASe, trois conditions doivent être vérifiées: * D'une part, les données à publier doivent être stockées dans une système gestionnaires de base de données relationnelles (parfois abrégé '''SGBDR''' ou '''RDBMS''' en anglais) compatible avec le langage '''SQL'''. Son modèle de données doit être bien documenté, ou, au minimum, présent à l'esprit d'un gestionnaire de données. Les bases de données non compatibles avec SQL ne sont pas supportées par BioCASe. Actuellement, le logiciel peut se connecter au bases de données suivantes: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * Pour installer BioCASe, vous devez avoir un logiciel de serveur Web (Apache ou Microsoft IIS) accessible en permanence depuis l'Internet. Cela peut être le serveur d'une institution académique ou scientifique, ou un service d'hébergement. Il est parfois pénible dans ce second cas d'obtenir les droits d'installation auprès d'un service d'hébergement. Une connexion commutée peer to peer ou 'Dial-up' ne fonctionnera pas. Notez qu'il est tout à fait possible d'installer un serveur Internet sur votre propre machine, isolé du reste de l'Internet, et de faire fonctionner BioCASe dessus à des fins d'entraînement, de formation ou de développement, . Il sera alors accessible localement pour l'ordinateur seul (via le nom http://localhost ou l'adresse IP réservée 127.0.0.1) * Vous devez avoir la possibilité d'installer Python sur le serveur web en question (si jamais ce n'est pas le cas) ainsi que des librairies ou paquets complémentaires Python. Cela peut être effectué relativement facilement par un informaticien ou quelqu'un ayant de bonnes connaissances techniques, mais assurez vous que cela ne pose pas problèmes par rapport aux procédures et règles techniques qui pourraient être en vigueur dans votre institution. == Etapes == Une fois que vous vous êtes assurez que le BioCASe Provider Software est un choix pertinent pour vos besoins ([[Special:Contact|contactez l'équipe BioCASe]] si vous souhaitez plus d'information avant de prendre une décision), vous devez exécuter les étapes suivantes: # Pensez à l'information que vous souhaitez publier sur le réseau. Cela implique d'avoir réfléchi et répondu à la question de la propriété des données et des termes d'utilisation, licences, copyright et restrictions dans l'utilisation qui s'appliquent à leur publication. Vous devriez aussi vous demander si certaines information ne devraient '''pas''' être publiées car cela pourrait avoir une conséquence sur des espèces menacées. une solution possible est de soustraire certaines occurrence du jeu de données publier, ou de rendre moins précise une information exacte sur la localité d'observation. # Préparez la base de données comme décrit dans le tutoriel [[Preparation]]. C'est à cette étape que doit être prise la décision de publier soit directement la base de données principale qui est utilisée pour les mises à jour, soit un 'snapshot' (c'est à dire une copie de tout ou une partie des données prise à un moment précis), de créer une table de métadonnées (informations sur la base de données elle-même) et, dans certains cas, de transformer la présentation des données. # Suivez le tutoriel d'[[Installation_(French)|Installation]] pour installer le Fournisseur de données BioCASe. # Configurez un jeu de données BioCASe viale logiciel et connectez-le à votre base de données. Configurez alors le jeu de données comme décrit dans le tutoriel [[DatasourceSetup]]. # Suivez le tutoriel [[ABCD2Mapping| ABCD2Mapping (pour l'instant en anglais)]] pour créer un mapping (un mapping est une correspondance entre la structure de votre bases de données et un schéma XML) pour le schéma de données (par exemple DarwinCore ou ABCD) que vous voulez utiliser. # Testez et débuggez votre web service BioCASe. Lisez le tutoriel [[Debugging]] pour plus d'informations à ce sujet. # Quand vous avez terminé avec l'installation et la configuration du BioCASE Provider Software, vous pouvez enregistrer le web service BioCASe auprès du réseau sur lequel vous souhaitez publier vos données, par exemple le [http://www.gbif.org GBIF]. Nous vous recommandons de contacter l'[[Special:Contact|équipe BioCASe]] pour vérifier au préalable votre web service. Ils pourront alors diagnostiquer lors de cette étape préalable des problèmes éventuels qui seraient plus compliqués à résoudre après l'enregistrement à un réseau. == Support == En cas de difficultés lors de l'installation et de la configuration, sous devriez: * jeter un oeil sur le [[FAQ]] qui contient peut-être une réponse à votre question * demander l'aide au staff technique d'un projet lié à BioCASe (comme CABIN ou OpenUp!) * demander l'aide d'autres utilisateurs de BioCASe * contacter l'[[Special:Contact|BioCASe team]], qui est plutôt anglophone. Décrivez le plus précisément possible votre environnement (système d'exploitation et système de bases de données utlisés), l'étape où le problème apparaît, et tous les messages d'erreurs ou informations techniques retourné par le logiciel. Il est recommendé d'envoyer des copies d'écran à vos interlocuteurs pour ce type de problème. b59af061b98b09a7b965eecb130e9e982e64a56b 517 516 2012-04-10T15:36:36Z FranckTheeten 15 /* Support */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' == Qu'est ce que le Logiciel Fournisseur de données BioCASe? == Le logiciel fournisseur de données BioCASe (abrégé '''BPS'' pour '''BioCASe Provider Software''' en anglais ) est un middleware, c'est à dire un logiciel situéà une position intermédiaire entre une base de données et une interface graphique. Il produit des données en XML, afin de les publier sur un réseau d'information scientifique. Après l'installation de BioCASe et sa configuration pour un type de base de données particulier, l'information publiée sera accessible sous la forme d'un web service de type BioCASe. Cela veut dire que l'information pourra être interrogée et recherchée avec des requêtes au format BioCASe, car BioCASe désigne également un protocole XML pour envoyer des messages à un web service. L'intérêt de ce type de logiciel est qu'il est "agnostique" par rapport au modèle de données de la base de données d'origine, c'est à dire que les requêtes envoyées au web services n'ont pas besoin de connaître la structure de la base de données pour fonctionner. Il peut être également utilisé avec n'importe quel type de schéma conceptuel de données en XML. Le BPS est un ensemble d'outils qui doivent être installée sur un serveur Web installé chez le fournisseur de données. Son composant central est le '''PyWrapper''', une interface XML/CGI, écrite en Python, pour se connecter à des bases de données . Il permet un accès standardisé (c'est à dire via des procédures uniformes qui peuvent être ainsi partagée par tous les acteurs d'un réseau d'information) à une large gamme de systèmes de base de données dont la structure est arbitraire (c'est à dire définie par le fournisseur de données seul). Un certain nombre d'outils est réuni autour du BPS pour configurer, tester et débugger une installation ou un web service BioCASe. Même si BioCASe peut être utilisé avec n'importe quel schéma conceptuel défini en XML, son domaine de prédilection est la publication de données d'occurrence provenant de bases de données contenant des spécimens ou des observations sur des réseaux d'information primaire sur la biodiversité, comme le [http://www.biocase.org réseau BioCASe] et le [http://www.gbif.org GBIF (Global Biodiversity Information Facility)]. == Prérequis pour utiliser le BPS == Avant d'utiliser le logiciel fournisseur de données BioCASe, trois conditions doivent être vérifiées: * D'une part, les données à publier doivent être stockées dans une système gestionnaires de base de données relationnelles (parfois abrégé '''SGBDR''' ou '''RDBMS''' en anglais) compatible avec le langage '''SQL'''. Son modèle de données doit être bien documenté, ou, au minimum, présent à l'esprit d'un gestionnaire de données. Les bases de données non compatibles avec SQL ne sont pas supportées par BioCASe. Actuellement, le logiciel peut se connecter au bases de données suivantes: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * Pour installer BioCASe, vous devez avoir un logiciel de serveur Web (Apache ou Microsoft IIS) accessible en permanence depuis l'Internet. Cela peut être le serveur d'une institution académique ou scientifique, ou un service d'hébergement. Il est parfois pénible dans ce second cas d'obtenir les droits d'installation auprès d'un service d'hébergement. Une connexion commutée peer to peer ou 'Dial-up' ne fonctionnera pas. Notez qu'il est tout à fait possible d'installer un serveur Internet sur votre propre machine, isolé du reste de l'Internet, et de faire fonctionner BioCASe dessus à des fins d'entraînement, de formation ou de développement, . Il sera alors accessible localement pour l'ordinateur seul (via le nom http://localhost ou l'adresse IP réservée 127.0.0.1) * Vous devez avoir la possibilité d'installer Python sur le serveur web en question (si jamais ce n'est pas le cas) ainsi que des librairies ou paquets complémentaires Python. Cela peut être effectué relativement facilement par un informaticien ou quelqu'un ayant de bonnes connaissances techniques, mais assurez vous que cela ne pose pas problèmes par rapport aux procédures et règles techniques qui pourraient être en vigueur dans votre institution. == Etapes == Une fois que vous vous êtes assurez que le BioCASe Provider Software est un choix pertinent pour vos besoins ([[Special:Contact|contactez l'équipe BioCASe]] si vous souhaitez plus d'information avant de prendre une décision), vous devez exécuter les étapes suivantes: # Pensez à l'information que vous souhaitez publier sur le réseau. Cela implique d'avoir réfléchi et répondu à la question de la propriété des données et des termes d'utilisation, licences, copyright et restrictions dans l'utilisation qui s'appliquent à leur publication. Vous devriez aussi vous demander si certaines information ne devraient '''pas''' être publiées car cela pourrait avoir une conséquence sur des espèces menacées. une solution possible est de soustraire certaines occurrence du jeu de données publier, ou de rendre moins précise une information exacte sur la localité d'observation. # Préparez la base de données comme décrit dans le tutoriel [[Preparation]]. C'est à cette étape que doit être prise la décision de publier soit directement la base de données principale qui est utilisée pour les mises à jour, soit un 'snapshot' (c'est à dire une copie de tout ou une partie des données prise à un moment précis), de créer une table de métadonnées (informations sur la base de données elle-même) et, dans certains cas, de transformer la présentation des données. # Suivez le tutoriel d'[[Installation_(French)|Installation]] pour installer le Fournisseur de données BioCASe. # Configurez un jeu de données BioCASe viale logiciel et connectez-le à votre base de données. Configurez alors le jeu de données comme décrit dans le tutoriel [[DatasourceSetup]]. # Suivez le tutoriel [[ABCD2Mapping| ABCD2Mapping (pour l'instant en anglais)]] pour créer un mapping (un mapping est une correspondance entre la structure de votre bases de données et un schéma XML) pour le schéma de données (par exemple DarwinCore ou ABCD) que vous voulez utiliser. # Testez et débuggez votre web service BioCASe. Lisez le tutoriel [[Debugging]] pour plus d'informations à ce sujet. # Quand vous avez terminé avec l'installation et la configuration du BioCASE Provider Software, vous pouvez enregistrer le web service BioCASe auprès du réseau sur lequel vous souhaitez publier vos données, par exemple le [http://www.gbif.org GBIF]. Nous vous recommandons de contacter l'[[Special:Contact|équipe BioCASe]] pour vérifier au préalable votre web service. Ils pourront alors diagnostiquer lors de cette étape préalable des problèmes éventuels qui seraient plus compliqués à résoudre après l'enregistrement à un réseau. == Support == En cas de difficultés lors de l'installation et de la configuration, vous devriez: * jeter un oeil sur le [[FAQ]] qui contient peut-être une réponse à votre question * demander l'aide au staff technique d'un projet lié à BioCASe (comme CABIN ou OpenUp!) * demander l'aide d'autres utilisateurs de BioCASe * contacter l'[[Special:Contact|BioCASe team]], qui est plutôt anglophone. Décrivez le plus précisément possible votre environnement (système d'exploitation et système de bases de données utlisés), l'étape où le problème apparaît, et tous les messages d'erreurs ou informations techniques retourné par le logiciel. Il est recommendé d'envoyer des copies d'écran à vos interlocuteurs pour ce type de problème. 1a278848ef623de4decb522e65d212c37d3c7236 518 517 2012-04-10T15:36:48Z FranckTheeten 15 /* Support */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' == Qu'est ce que le Logiciel Fournisseur de données BioCASe? == Le logiciel fournisseur de données BioCASe (abrégé '''BPS'' pour '''BioCASe Provider Software''' en anglais ) est un middleware, c'est à dire un logiciel situéà une position intermédiaire entre une base de données et une interface graphique. Il produit des données en XML, afin de les publier sur un réseau d'information scientifique. Après l'installation de BioCASe et sa configuration pour un type de base de données particulier, l'information publiée sera accessible sous la forme d'un web service de type BioCASe. Cela veut dire que l'information pourra être interrogée et recherchée avec des requêtes au format BioCASe, car BioCASe désigne également un protocole XML pour envoyer des messages à un web service. L'intérêt de ce type de logiciel est qu'il est "agnostique" par rapport au modèle de données de la base de données d'origine, c'est à dire que les requêtes envoyées au web services n'ont pas besoin de connaître la structure de la base de données pour fonctionner. Il peut être également utilisé avec n'importe quel type de schéma conceptuel de données en XML. Le BPS est un ensemble d'outils qui doivent être installée sur un serveur Web installé chez le fournisseur de données. Son composant central est le '''PyWrapper''', une interface XML/CGI, écrite en Python, pour se connecter à des bases de données . Il permet un accès standardisé (c'est à dire via des procédures uniformes qui peuvent être ainsi partagée par tous les acteurs d'un réseau d'information) à une large gamme de systèmes de base de données dont la structure est arbitraire (c'est à dire définie par le fournisseur de données seul). Un certain nombre d'outils est réuni autour du BPS pour configurer, tester et débugger une installation ou un web service BioCASe. Même si BioCASe peut être utilisé avec n'importe quel schéma conceptuel défini en XML, son domaine de prédilection est la publication de données d'occurrence provenant de bases de données contenant des spécimens ou des observations sur des réseaux d'information primaire sur la biodiversité, comme le [http://www.biocase.org réseau BioCASe] et le [http://www.gbif.org GBIF (Global Biodiversity Information Facility)]. == Prérequis pour utiliser le BPS == Avant d'utiliser le logiciel fournisseur de données BioCASe, trois conditions doivent être vérifiées: * D'une part, les données à publier doivent être stockées dans une système gestionnaires de base de données relationnelles (parfois abrégé '''SGBDR''' ou '''RDBMS''' en anglais) compatible avec le langage '''SQL'''. Son modèle de données doit être bien documenté, ou, au minimum, présent à l'esprit d'un gestionnaire de données. Les bases de données non compatibles avec SQL ne sont pas supportées par BioCASe. Actuellement, le logiciel peut se connecter au bases de données suivantes: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * Pour installer BioCASe, vous devez avoir un logiciel de serveur Web (Apache ou Microsoft IIS) accessible en permanence depuis l'Internet. Cela peut être le serveur d'une institution académique ou scientifique, ou un service d'hébergement. Il est parfois pénible dans ce second cas d'obtenir les droits d'installation auprès d'un service d'hébergement. Une connexion commutée peer to peer ou 'Dial-up' ne fonctionnera pas. Notez qu'il est tout à fait possible d'installer un serveur Internet sur votre propre machine, isolé du reste de l'Internet, et de faire fonctionner BioCASe dessus à des fins d'entraînement, de formation ou de développement, . Il sera alors accessible localement pour l'ordinateur seul (via le nom http://localhost ou l'adresse IP réservée 127.0.0.1) * Vous devez avoir la possibilité d'installer Python sur le serveur web en question (si jamais ce n'est pas le cas) ainsi que des librairies ou paquets complémentaires Python. Cela peut être effectué relativement facilement par un informaticien ou quelqu'un ayant de bonnes connaissances techniques, mais assurez vous que cela ne pose pas problèmes par rapport aux procédures et règles techniques qui pourraient être en vigueur dans votre institution. == Etapes == Une fois que vous vous êtes assurez que le BioCASe Provider Software est un choix pertinent pour vos besoins ([[Special:Contact|contactez l'équipe BioCASe]] si vous souhaitez plus d'information avant de prendre une décision), vous devez exécuter les étapes suivantes: # Pensez à l'information que vous souhaitez publier sur le réseau. Cela implique d'avoir réfléchi et répondu à la question de la propriété des données et des termes d'utilisation, licences, copyright et restrictions dans l'utilisation qui s'appliquent à leur publication. Vous devriez aussi vous demander si certaines information ne devraient '''pas''' être publiées car cela pourrait avoir une conséquence sur des espèces menacées. une solution possible est de soustraire certaines occurrence du jeu de données publier, ou de rendre moins précise une information exacte sur la localité d'observation. # Préparez la base de données comme décrit dans le tutoriel [[Preparation]]. C'est à cette étape que doit être prise la décision de publier soit directement la base de données principale qui est utilisée pour les mises à jour, soit un 'snapshot' (c'est à dire une copie de tout ou une partie des données prise à un moment précis), de créer une table de métadonnées (informations sur la base de données elle-même) et, dans certains cas, de transformer la présentation des données. # Suivez le tutoriel d'[[Installation_(French)|Installation]] pour installer le Fournisseur de données BioCASe. # Configurez un jeu de données BioCASe viale logiciel et connectez-le à votre base de données. Configurez alors le jeu de données comme décrit dans le tutoriel [[DatasourceSetup]]. # Suivez le tutoriel [[ABCD2Mapping| ABCD2Mapping (pour l'instant en anglais)]] pour créer un mapping (un mapping est une correspondance entre la structure de votre bases de données et un schéma XML) pour le schéma de données (par exemple DarwinCore ou ABCD) que vous voulez utiliser. # Testez et débuggez votre web service BioCASe. Lisez le tutoriel [[Debugging]] pour plus d'informations à ce sujet. # Quand vous avez terminé avec l'installation et la configuration du BioCASE Provider Software, vous pouvez enregistrer le web service BioCASe auprès du réseau sur lequel vous souhaitez publier vos données, par exemple le [http://www.gbif.org GBIF]. Nous vous recommandons de contacter l'[[Special:Contact|équipe BioCASe]] pour vérifier au préalable votre web service. Ils pourront alors diagnostiquer lors de cette étape préalable des problèmes éventuels qui seraient plus compliqués à résoudre après l'enregistrement à un réseau. == Support == En cas de difficultés lors de l'installation et de la configuration, vous devriez: * jeter un œil sur le [[FAQ]] qui contient peut-être une réponse à votre question * demander l'aide au staff technique d'un projet lié à BioCASe (comme CABIN ou OpenUp!) * demander l'aide d'autres utilisateurs de BioCASe * contacter l'[[Special:Contact|BioCASe team]], qui est plutôt anglophone. Décrivez le plus précisément possible votre environnement (système d'exploitation et système de bases de données utlisés), l'étape où le problème apparaît, et tous les messages d'erreurs ou informations techniques retourné par le logiciel. Il est recommendé d'envoyer des copies d'écran à vos interlocuteurs pour ce type de problème. 0b8ba3d69a0d64a98c1314a82e763b7711bc9aff 0 93 519 518 2012-04-10T15:38:11Z FranckTheeten 15 /* Support */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' == Qu'est ce que le Logiciel Fournisseur de données BioCASe? == Le logiciel fournisseur de données BioCASe (abrégé '''BPS'' pour '''BioCASe Provider Software''' en anglais ) est un middleware, c'est à dire un logiciel situéà une position intermédiaire entre une base de données et une interface graphique. Il produit des données en XML, afin de les publier sur un réseau d'information scientifique. Après l'installation de BioCASe et sa configuration pour un type de base de données particulier, l'information publiée sera accessible sous la forme d'un web service de type BioCASe. Cela veut dire que l'information pourra être interrogée et recherchée avec des requêtes au format BioCASe, car BioCASe désigne également un protocole XML pour envoyer des messages à un web service. L'intérêt de ce type de logiciel est qu'il est "agnostique" par rapport au modèle de données de la base de données d'origine, c'est à dire que les requêtes envoyées au web services n'ont pas besoin de connaître la structure de la base de données pour fonctionner. Il peut être également utilisé avec n'importe quel type de schéma conceptuel de données en XML. Le BPS est un ensemble d'outils qui doivent être installée sur un serveur Web installé chez le fournisseur de données. Son composant central est le '''PyWrapper''', une interface XML/CGI, écrite en Python, pour se connecter à des bases de données . Il permet un accès standardisé (c'est à dire via des procédures uniformes qui peuvent être ainsi partagée par tous les acteurs d'un réseau d'information) à une large gamme de systèmes de base de données dont la structure est arbitraire (c'est à dire définie par le fournisseur de données seul). Un certain nombre d'outils est réuni autour du BPS pour configurer, tester et débugger une installation ou un web service BioCASe. Même si BioCASe peut être utilisé avec n'importe quel schéma conceptuel défini en XML, son domaine de prédilection est la publication de données d'occurrence provenant de bases de données contenant des spécimens ou des observations sur des réseaux d'information primaire sur la biodiversité, comme le [http://www.biocase.org réseau BioCASe] et le [http://www.gbif.org GBIF (Global Biodiversity Information Facility)]. == Prérequis pour utiliser le BPS == Avant d'utiliser le logiciel fournisseur de données BioCASe, trois conditions doivent être vérifiées: * D'une part, les données à publier doivent être stockées dans une système gestionnaires de base de données relationnelles (parfois abrégé '''SGBDR''' ou '''RDBMS''' en anglais) compatible avec le langage '''SQL'''. Son modèle de données doit être bien documenté, ou, au minimum, présent à l'esprit d'un gestionnaire de données. Les bases de données non compatibles avec SQL ne sont pas supportées par BioCASe. Actuellement, le logiciel peut se connecter au bases de données suivantes: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * Pour installer BioCASe, vous devez avoir un logiciel de serveur Web (Apache ou Microsoft IIS) accessible en permanence depuis l'Internet. Cela peut être le serveur d'une institution académique ou scientifique, ou un service d'hébergement. Il est parfois pénible dans ce second cas d'obtenir les droits d'installation auprès d'un service d'hébergement. Une connexion commutée peer to peer ou 'Dial-up' ne fonctionnera pas. Notez qu'il est tout à fait possible d'installer un serveur Internet sur votre propre machine, isolé du reste de l'Internet, et de faire fonctionner BioCASe dessus à des fins d'entraînement, de formation ou de développement, . Il sera alors accessible localement pour l'ordinateur seul (via le nom http://localhost ou l'adresse IP réservée 127.0.0.1) * Vous devez avoir la possibilité d'installer Python sur le serveur web en question (si jamais ce n'est pas le cas) ainsi que des librairies ou paquets complémentaires Python. Cela peut être effectué relativement facilement par un informaticien ou quelqu'un ayant de bonnes connaissances techniques, mais assurez vous que cela ne pose pas problèmes par rapport aux procédures et règles techniques qui pourraient être en vigueur dans votre institution. == Etapes == Une fois que vous vous êtes assurez que le BioCASe Provider Software est un choix pertinent pour vos besoins ([[Special:Contact|contactez l'équipe BioCASe]] si vous souhaitez plus d'information avant de prendre une décision), vous devez exécuter les étapes suivantes: # Pensez à l'information que vous souhaitez publier sur le réseau. Cela implique d'avoir réfléchi et répondu à la question de la propriété des données et des termes d'utilisation, licences, copyright et restrictions dans l'utilisation qui s'appliquent à leur publication. Vous devriez aussi vous demander si certaines information ne devraient '''pas''' être publiées car cela pourrait avoir une conséquence sur des espèces menacées. une solution possible est de soustraire certaines occurrence du jeu de données publier, ou de rendre moins précise une information exacte sur la localité d'observation. # Préparez la base de données comme décrit dans le tutoriel [[Preparation]]. C'est à cette étape que doit être prise la décision de publier soit directement la base de données principale qui est utilisée pour les mises à jour, soit un 'snapshot' (c'est à dire une copie de tout ou une partie des données prise à un moment précis), de créer une table de métadonnées (informations sur la base de données elle-même) et, dans certains cas, de transformer la présentation des données. # Suivez le tutoriel d'[[Installation_(French)|Installation]] pour installer le Fournisseur de données BioCASe. # Configurez un jeu de données BioCASe viale logiciel et connectez-le à votre base de données. Configurez alors le jeu de données comme décrit dans le tutoriel [[DatasourceSetup]]. # Suivez le tutoriel [[ABCD2Mapping| ABCD2Mapping (pour l'instant en anglais)]] pour créer un mapping (un mapping est une correspondance entre la structure de votre bases de données et un schéma XML) pour le schéma de données (par exemple DarwinCore ou ABCD) que vous voulez utiliser. # Testez et débuggez votre web service BioCASe. Lisez le tutoriel [[Debugging]] pour plus d'informations à ce sujet. # Quand vous avez terminé avec l'installation et la configuration du BioCASE Provider Software, vous pouvez enregistrer le web service BioCASe auprès du réseau sur lequel vous souhaitez publier vos données, par exemple le [http://www.gbif.org GBIF]. Nous vous recommandons de contacter l'[[Special:Contact|équipe BioCASe]] pour vérifier au préalable votre web service. Ils pourront alors diagnostiquer lors de cette étape préalable des problèmes éventuels qui seraient plus compliqués à résoudre après l'enregistrement à un réseau. == Support == En cas de difficultés lors de l'installation et de la configuration, vous devriez: * jeter un œil sur le [[FAQ]] qui contient peut-être une réponse à votre question * demander l'aide au staff technique d'un projet lié à BioCASe (comme CABIN ou OpenUp!) * demander l'aide d'autres utilisateurs de BioCASe * contacter l'[[Special:Contact|équipe BioCASe]](qui est plutôt anglophone mais peut si nécessaire transmettre des questions en français à des techniciens francophones). Décrivez le plus précisément possible votre environnement (système d'exploitation et système de bases de données utlisés), l'étape où le problème apparaît, et tous les messages d'erreurs ou informations techniques retourné par le logiciel. Il est recommandé d'envoyer des '''copies d'écran''' à vos interlocuteurs pour faciltier l'identification du problème. 2d970cd8efcf88f26c3391032c7bb3ebd60875c8 520 519 2012-04-10T15:38:25Z FranckTheeten 15 /* Support */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' == Qu'est ce que le Logiciel Fournisseur de données BioCASe? == Le logiciel fournisseur de données BioCASe (abrégé '''BPS'' pour '''BioCASe Provider Software''' en anglais ) est un middleware, c'est à dire un logiciel situéà une position intermédiaire entre une base de données et une interface graphique. Il produit des données en XML, afin de les publier sur un réseau d'information scientifique. Après l'installation de BioCASe et sa configuration pour un type de base de données particulier, l'information publiée sera accessible sous la forme d'un web service de type BioCASe. Cela veut dire que l'information pourra être interrogée et recherchée avec des requêtes au format BioCASe, car BioCASe désigne également un protocole XML pour envoyer des messages à un web service. L'intérêt de ce type de logiciel est qu'il est "agnostique" par rapport au modèle de données de la base de données d'origine, c'est à dire que les requêtes envoyées au web services n'ont pas besoin de connaître la structure de la base de données pour fonctionner. Il peut être également utilisé avec n'importe quel type de schéma conceptuel de données en XML. Le BPS est un ensemble d'outils qui doivent être installée sur un serveur Web installé chez le fournisseur de données. Son composant central est le '''PyWrapper''', une interface XML/CGI, écrite en Python, pour se connecter à des bases de données . Il permet un accès standardisé (c'est à dire via des procédures uniformes qui peuvent être ainsi partagée par tous les acteurs d'un réseau d'information) à une large gamme de systèmes de base de données dont la structure est arbitraire (c'est à dire définie par le fournisseur de données seul). Un certain nombre d'outils est réuni autour du BPS pour configurer, tester et débugger une installation ou un web service BioCASe. Même si BioCASe peut être utilisé avec n'importe quel schéma conceptuel défini en XML, son domaine de prédilection est la publication de données d'occurrence provenant de bases de données contenant des spécimens ou des observations sur des réseaux d'information primaire sur la biodiversité, comme le [http://www.biocase.org réseau BioCASe] et le [http://www.gbif.org GBIF (Global Biodiversity Information Facility)]. == Prérequis pour utiliser le BPS == Avant d'utiliser le logiciel fournisseur de données BioCASe, trois conditions doivent être vérifiées: * D'une part, les données à publier doivent être stockées dans une système gestionnaires de base de données relationnelles (parfois abrégé '''SGBDR''' ou '''RDBMS''' en anglais) compatible avec le langage '''SQL'''. Son modèle de données doit être bien documenté, ou, au minimum, présent à l'esprit d'un gestionnaire de données. Les bases de données non compatibles avec SQL ne sont pas supportées par BioCASe. Actuellement, le logiciel peut se connecter au bases de données suivantes: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * Pour installer BioCASe, vous devez avoir un logiciel de serveur Web (Apache ou Microsoft IIS) accessible en permanence depuis l'Internet. Cela peut être le serveur d'une institution académique ou scientifique, ou un service d'hébergement. Il est parfois pénible dans ce second cas d'obtenir les droits d'installation auprès d'un service d'hébergement. Une connexion commutée peer to peer ou 'Dial-up' ne fonctionnera pas. Notez qu'il est tout à fait possible d'installer un serveur Internet sur votre propre machine, isolé du reste de l'Internet, et de faire fonctionner BioCASe dessus à des fins d'entraînement, de formation ou de développement, . Il sera alors accessible localement pour l'ordinateur seul (via le nom http://localhost ou l'adresse IP réservée 127.0.0.1) * Vous devez avoir la possibilité d'installer Python sur le serveur web en question (si jamais ce n'est pas le cas) ainsi que des librairies ou paquets complémentaires Python. Cela peut être effectué relativement facilement par un informaticien ou quelqu'un ayant de bonnes connaissances techniques, mais assurez vous que cela ne pose pas problèmes par rapport aux procédures et règles techniques qui pourraient être en vigueur dans votre institution. == Etapes == Une fois que vous vous êtes assurez que le BioCASe Provider Software est un choix pertinent pour vos besoins ([[Special:Contact|contactez l'équipe BioCASe]] si vous souhaitez plus d'information avant de prendre une décision), vous devez exécuter les étapes suivantes: # Pensez à l'information que vous souhaitez publier sur le réseau. Cela implique d'avoir réfléchi et répondu à la question de la propriété des données et des termes d'utilisation, licences, copyright et restrictions dans l'utilisation qui s'appliquent à leur publication. Vous devriez aussi vous demander si certaines information ne devraient '''pas''' être publiées car cela pourrait avoir une conséquence sur des espèces menacées. une solution possible est de soustraire certaines occurrence du jeu de données publier, ou de rendre moins précise une information exacte sur la localité d'observation. # Préparez la base de données comme décrit dans le tutoriel [[Preparation]]. C'est à cette étape que doit être prise la décision de publier soit directement la base de données principale qui est utilisée pour les mises à jour, soit un 'snapshot' (c'est à dire une copie de tout ou une partie des données prise à un moment précis), de créer une table de métadonnées (informations sur la base de données elle-même) et, dans certains cas, de transformer la présentation des données. # Suivez le tutoriel d'[[Installation_(French)|Installation]] pour installer le Fournisseur de données BioCASe. # Configurez un jeu de données BioCASe viale logiciel et connectez-le à votre base de données. Configurez alors le jeu de données comme décrit dans le tutoriel [[DatasourceSetup]]. # Suivez le tutoriel [[ABCD2Mapping| ABCD2Mapping (pour l'instant en anglais)]] pour créer un mapping (un mapping est une correspondance entre la structure de votre bases de données et un schéma XML) pour le schéma de données (par exemple DarwinCore ou ABCD) que vous voulez utiliser. # Testez et débuggez votre web service BioCASe. Lisez le tutoriel [[Debugging]] pour plus d'informations à ce sujet. # Quand vous avez terminé avec l'installation et la configuration du BioCASE Provider Software, vous pouvez enregistrer le web service BioCASe auprès du réseau sur lequel vous souhaitez publier vos données, par exemple le [http://www.gbif.org GBIF]. Nous vous recommandons de contacter l'[[Special:Contact|équipe BioCASe]] pour vérifier au préalable votre web service. Ils pourront alors diagnostiquer lors de cette étape préalable des problèmes éventuels qui seraient plus compliqués à résoudre après l'enregistrement à un réseau. == Support == En cas de difficultés lors de l'installation et de la configuration, vous devriez: * jeter un œil sur le [[FAQ]] qui contient peut-être une réponse à votre question * demander l'aide au staff technique d'un projet lié à BioCASe (comme CABIN ou OpenUp!) * demander l'aide d'autres utilisateurs de BioCASe * contacter l'[[Special:Contact|équipe BioCASe]](qui est plutôt anglophone mais peut si nécessaire transmettre des questions en français à des techniciens francophones). Décrivez le plus précisément possible votre environnement (système d'exploitation et système de bases de données utlisés), l'étape où le problème apparaît, et tous les messages d'erreurs ou informations techniques retourné par le logiciel. Il est recommandé d'envoyer des '''copies d'écran''' à vos interlocuteurs pour faciliter l'identification du problème. b7bae62a0e9075e4f47c45853922d3ff09c39664 551 520 2012-04-10T16:18:10Z FranckTheeten 15 /* Prérequis pour utiliser le BPS */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' == Qu'est ce que le Logiciel Fournisseur de données BioCASe? == Le logiciel fournisseur de données BioCASe (abrégé '''BPS'' pour '''BioCASe Provider Software''' en anglais ) est un middleware, c'est à dire un logiciel situéà une position intermédiaire entre une base de données et une interface graphique. Il produit des données en XML, afin de les publier sur un réseau d'information scientifique. Après l'installation de BioCASe et sa configuration pour un type de base de données particulier, l'information publiée sera accessible sous la forme d'un web service de type BioCASe. Cela veut dire que l'information pourra être interrogée et recherchée avec des requêtes au format BioCASe, car BioCASe désigne également un protocole XML pour envoyer des messages à un web service. L'intérêt de ce type de logiciel est qu'il est "agnostique" par rapport au modèle de données de la base de données d'origine, c'est à dire que les requêtes envoyées au web services n'ont pas besoin de connaître la structure de la base de données pour fonctionner. Il peut être également utilisé avec n'importe quel type de schéma conceptuel de données en XML. Le BPS est un ensemble d'outils qui doivent être installée sur un serveur Web installé chez le fournisseur de données. Son composant central est le '''PyWrapper''', une interface XML/CGI, écrite en Python, pour se connecter à des bases de données . Il permet un accès standardisé (c'est à dire via des procédures uniformes qui peuvent être ainsi partagée par tous les acteurs d'un réseau d'information) à une large gamme de systèmes de base de données dont la structure est arbitraire (c'est à dire définie par le fournisseur de données seul). Un certain nombre d'outils est réuni autour du BPS pour configurer, tester et débugger une installation ou un web service BioCASe. Même si BioCASe peut être utilisé avec n'importe quel schéma conceptuel défini en XML, son domaine de prédilection est la publication de données d'occurrence provenant de bases de données contenant des spécimens ou des observations sur des réseaux d'information primaire sur la biodiversité, comme le [http://www.biocase.org réseau BioCASe] et le [http://www.gbif.org GBIF (Global Biodiversity Information Facility)]. == Prérequis pour utiliser le BPS == Avant d'utiliser le logiciel fournisseur de données BioCASe, trois conditions doivent être vérifiées: * D'une part, les données à publier doivent être stockées dans une système gestionnaires de base de données relationnelles (parfois abrégé '''SGBDR''' ou '''RDBMS''' en anglais) compatible avec le langage '''SQL'''. Son modèle de données doit être bien documenté, ou, au minimum, présent à l'esprit d'un gestionnaire de données. Les bases de données non compatibles avec SQL ne sont pas supportées par BioCASe. Actuellement, le logiciel peut se connecter au bases de données suivantes: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * Pour installer BioCASe, vous devez avoir un logiciel de serveur Web (Apache ou Microsoft IIS) accessible en permanence depuis l'Internet. Cela peut être le serveur d'une institution académique ou scientifique, ou un service d'hébergement. Il est parfois pénible dans ce second cas d'obtenir les droits d'installation auprès d'un service d'hébergement. Une connexion commutée peer to peer ou 'Dial-up' ne fonctionnera pas. Notez qu'il est tout à fait possible d'installer un serveur Internet sur votre propre machine, isolé du reste de l'Internet, et de faire fonctionner BioCASe dessus à des fins d'entraînement, de formation ou de développement. Il sera alors accessible localement pour l'ordinateur seul (via le nom http://localhost ou l'adresse IP réservée 127.0.0.1) * Vous devez avoir la possibilité d'installer Python sur le serveur web en question (si jamais ce n'est pas le cas) ainsi que des librairies ou paquets complémentaires Python. Cela peut être effectué relativement facilement par un informaticien ou quelqu'un ayant de bonnes connaissances techniques, mais assurez vous que cela ne pose pas problèmes par rapport aux procédures et règles techniques qui pourraient être en vigueur dans votre institution. == Etapes == Une fois que vous vous êtes assurez que le BioCASe Provider Software est un choix pertinent pour vos besoins ([[Special:Contact|contactez l'équipe BioCASe]] si vous souhaitez plus d'information avant de prendre une décision), vous devez exécuter les étapes suivantes: # Pensez à l'information que vous souhaitez publier sur le réseau. Cela implique d'avoir réfléchi et répondu à la question de la propriété des données et des termes d'utilisation, licences, copyright et restrictions dans l'utilisation qui s'appliquent à leur publication. Vous devriez aussi vous demander si certaines information ne devraient '''pas''' être publiées car cela pourrait avoir une conséquence sur des espèces menacées. une solution possible est de soustraire certaines occurrence du jeu de données publier, ou de rendre moins précise une information exacte sur la localité d'observation. # Préparez la base de données comme décrit dans le tutoriel [[Preparation]]. C'est à cette étape que doit être prise la décision de publier soit directement la base de données principale qui est utilisée pour les mises à jour, soit un 'snapshot' (c'est à dire une copie de tout ou une partie des données prise à un moment précis), de créer une table de métadonnées (informations sur la base de données elle-même) et, dans certains cas, de transformer la présentation des données. # Suivez le tutoriel d'[[Installation_(French)|Installation]] pour installer le Fournisseur de données BioCASe. # Configurez un jeu de données BioCASe viale logiciel et connectez-le à votre base de données. Configurez alors le jeu de données comme décrit dans le tutoriel [[DatasourceSetup]]. # Suivez le tutoriel [[ABCD2Mapping| ABCD2Mapping (pour l'instant en anglais)]] pour créer un mapping (un mapping est une correspondance entre la structure de votre bases de données et un schéma XML) pour le schéma de données (par exemple DarwinCore ou ABCD) que vous voulez utiliser. # Testez et débuggez votre web service BioCASe. Lisez le tutoriel [[Debugging]] pour plus d'informations à ce sujet. # Quand vous avez terminé avec l'installation et la configuration du BioCASE Provider Software, vous pouvez enregistrer le web service BioCASe auprès du réseau sur lequel vous souhaitez publier vos données, par exemple le [http://www.gbif.org GBIF]. Nous vous recommandons de contacter l'[[Special:Contact|équipe BioCASe]] pour vérifier au préalable votre web service. Ils pourront alors diagnostiquer lors de cette étape préalable des problèmes éventuels qui seraient plus compliqués à résoudre après l'enregistrement à un réseau. == Support == En cas de difficultés lors de l'installation et de la configuration, vous devriez: * jeter un œil sur le [[FAQ]] qui contient peut-être une réponse à votre question * demander l'aide au staff technique d'un projet lié à BioCASe (comme CABIN ou OpenUp!) * demander l'aide d'autres utilisateurs de BioCASe * contacter l'[[Special:Contact|équipe BioCASe]](qui est plutôt anglophone mais peut si nécessaire transmettre des questions en français à des techniciens francophones). Décrivez le plus précisément possible votre environnement (système d'exploitation et système de bases de données utlisés), l'étape où le problème apparaît, et tous les messages d'erreurs ou informations techniques retourné par le logiciel. Il est recommandé d'envoyer des '''copies d'écran''' à vos interlocuteurs pour faciliter l'identification du problème. d1ba98ffcf2846a72d70a18b9db3bbf110f5d2c8 0 94 521 498 2012-04-10T15:39:14Z FranckTheeten 15 /* Exécution du script d'installation */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes en fonction de votre degré de familiarité avec Python et Apache. Si vous êtes un débutant, veuillez lire le Guide du Débutant [[BeginnersGuide|Beginner’s Guide]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit soit être à l'extérieur d'un pare-feu institutionnel ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de satisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine à la base de données à publier - soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL Server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fonctionner également avec les versions 2.6 et 2.7, la version recommandée est de 2.5 Python (en fait la dernière sous-version stable de la branche 2.5: 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le bib site Web (par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous faites l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier Program Files par défaut. En effet à cet endroit le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir du Subversion du BGBM (Subversion ou SVN est un logiciel qui permet à plusieurs programmeurs de travailler en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la documentation des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (pré-installé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS P Oar exemple si votre provider est installé à l'emplacement tapez ''D:\\biocase''<pre>D:</pre><pre>cd D://biocase</pre>. Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Exemple sous Windows et DOS, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python donnée dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de interpréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecture depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur et permettre au modne extérieur de voir vos fichiers de configuration (qui incluent des mots de passe). Ainsi si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre server est ''d:\biocase\www' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache. Si vous installer BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez en informer l'administrateur avant de modifier ce fichier! [[image:instHttpdConf.png|480px]] Sauvez le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utlisez IIS vous devez créer d'abord un répertoire virtuel pour le répertoire physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'), utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier .cgi and .py avec le chemin de votre fichier exécutable Python. Vous devriez aussi aussi ajouter ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmons de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'accueil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du logiciel utilisé pour la base de données à publier et des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'accueil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes), qui indique également si cette librairie est présente sur le système et, si ou, dans quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci: [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe proprement dite sont déjà inclues dans le fichier d'archive que vous avez installé ou bien le répertoire Subversion d'où vous êtes partis. La librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos don"nes sur le Web) – ainsi si vous n'envisagez pas de l'utiliser il n'est pas nécessaire de l'installer. Sinon cliquez sur le lien 'Download', trouvez le fichier d'installation qui convient à votre système d'exploitation et installez 4SuiteXML à partir du fichier téléchargé. Néanmoins, Pour chaque type de base de données (SGBDR ou Système Gestionnaire be base de données Relationnelle dans le jargon), vous devrez installer le package ou driver Python correspondant. Ce driver permet à un programme Python de se connecter à une base de donénes . Il doit être installé et configuré non pas au niveau de BioCASe, mais à celui de l'environnement Python; mais néanmoins le BPS peut vous guider dans leur installation. Si votre collection est par exemple stockée dans une base de données MySQL, cliquez sur le lien pour télécharger le module MySQL, trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraires dans votre navigateur. La ligne de la liste qui correspond au driver devrait passer de l'orange pour 'non installé' à une indication verte mentionnant la version de la libraire. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utilisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessire d'installer Graphviz. Dans le cas contraire nous vous recommandons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquat, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les pramètres du BPS. A la fin de la procédure de configuration, vous devriez changer le mot de passe. Retournez à l'écran d'accueil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la magde Configuration dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation (nosu supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''). A la quatrième ligne, vous pouvez encoder un nouveau mot de passe. Soyez précautionneux en le tapant (si jamais vous l'oubliez, il est toujours possible de le réintialiser en modifiant directement le fichier de configuration). Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers binaires, car Graphviz n'est pas une librairie Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sut votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary</small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''), sous Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connection à une base de donénes), vous devriez vous assurer que tout est installé correctement: # Taper ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à votre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modificables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les Logs (fichiers qui enregistrent les opérations sur la machine) et la configuration. Sinon, le chemin vers le fichier posent problème sera montré –. Veuillez vous assurer que Python et.ou Apache peuvent écrire sur ce fichier (par exemple sur Ubuntu où Apache correspond à un compte utlsiateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre>). Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connexion via la librairie ODBC) et que vous ne souhaitez pas utiliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] 1a54e1a585a1e0f5256ae2878b8549fd5c162eaa 522 521 2012-04-10T15:39:49Z FranckTheeten 15 /* Exécution du script d'installation */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes en fonction de votre degré de familiarité avec Python et Apache. Si vous êtes un débutant, veuillez lire le Guide du Débutant [[BeginnersGuide|Beginner’s Guide]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit soit être à l'extérieur d'un pare-feu institutionnel ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de satisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine à la base de données à publier - soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL Server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fonctionner également avec les versions 2.6 et 2.7, la version recommandée est de 2.5 Python (en fait la dernière sous-version stable de la branche 2.5: 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le bib site Web (par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous faites l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier Program Files par défaut. En effet à cet endroit le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir du Subversion du BGBM (Subversion ou SVN est un logiciel qui permet à plusieurs programmeurs de travailler en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la documentation des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (pré-installé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS P Oar exemple si votre provider est installé à l'emplacement ''D:\\biocase'' tapez <pre>D:</pre><pre>cd D://biocase</pre> Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Exemple sous Windows et DOS, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python donnée dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de interpréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecture depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur et permettre au modne extérieur de voir vos fichiers de configuration (qui incluent des mots de passe). Ainsi si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre server est ''d:\biocase\www' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache. Si vous installer BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez en informer l'administrateur avant de modifier ce fichier! [[image:instHttpdConf.png|480px]] Sauvez le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utlisez IIS vous devez créer d'abord un répertoire virtuel pour le répertoire physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'), utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier .cgi and .py avec le chemin de votre fichier exécutable Python. Vous devriez aussi aussi ajouter ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmons de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'accueil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du logiciel utilisé pour la base de données à publier et des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'accueil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes), qui indique également si cette librairie est présente sur le système et, si ou, dans quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci: [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe proprement dite sont déjà inclues dans le fichier d'archive que vous avez installé ou bien le répertoire Subversion d'où vous êtes partis. La librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos don"nes sur le Web) – ainsi si vous n'envisagez pas de l'utiliser il n'est pas nécessaire de l'installer. Sinon cliquez sur le lien 'Download', trouvez le fichier d'installation qui convient à votre système d'exploitation et installez 4SuiteXML à partir du fichier téléchargé. Néanmoins, Pour chaque type de base de données (SGBDR ou Système Gestionnaire be base de données Relationnelle dans le jargon), vous devrez installer le package ou driver Python correspondant. Ce driver permet à un programme Python de se connecter à une base de donénes . Il doit être installé et configuré non pas au niveau de BioCASe, mais à celui de l'environnement Python; mais néanmoins le BPS peut vous guider dans leur installation. Si votre collection est par exemple stockée dans une base de données MySQL, cliquez sur le lien pour télécharger le module MySQL, trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraires dans votre navigateur. La ligne de la liste qui correspond au driver devrait passer de l'orange pour 'non installé' à une indication verte mentionnant la version de la libraire. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utilisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessire d'installer Graphviz. Dans le cas contraire nous vous recommandons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquat, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les pramètres du BPS. A la fin de la procédure de configuration, vous devriez changer le mot de passe. Retournez à l'écran d'accueil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la magde Configuration dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation (nosu supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''). A la quatrième ligne, vous pouvez encoder un nouveau mot de passe. Soyez précautionneux en le tapant (si jamais vous l'oubliez, il est toujours possible de le réintialiser en modifiant directement le fichier de configuration). Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers binaires, car Graphviz n'est pas une librairie Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sut votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary</small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''), sous Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connection à une base de donénes), vous devriez vous assurer que tout est installé correctement: # Taper ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à votre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modificables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les Logs (fichiers qui enregistrent les opérations sur la machine) et la configuration. Sinon, le chemin vers le fichier posent problème sera montré –. Veuillez vous assurer que Python et.ou Apache peuvent écrire sur ce fichier (par exemple sur Ubuntu où Apache correspond à un compte utlsiateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre>). Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connexion via la librairie ODBC) et que vous ne souhaitez pas utiliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] c49cc57a8f8b186da97ca6d1333df73340dbfdd9 523 522 2012-04-10T15:40:36Z FranckTheeten 15 /* Exécution du script d'installation */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes en fonction de votre degré de familiarité avec Python et Apache. Si vous êtes un débutant, veuillez lire le Guide du Débutant [[BeginnersGuide|Beginner’s Guide]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit soit être à l'extérieur d'un pare-feu institutionnel ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de satisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine à la base de données à publier - soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL Server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fonctionner également avec les versions 2.6 et 2.7, la version recommandée est de 2.5 Python (en fait la dernière sous-version stable de la branche 2.5: 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le bib site Web (par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous faites l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier Program Files par défaut. En effet à cet endroit le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir du Subversion du BGBM (Subversion ou SVN est un logiciel qui permet à plusieurs programmeurs de travailler en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la documentation des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (pré-installé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS P Oar exemple si votre provider est installé à l'emplacement ''D:\\biocase'' tapez <pre>D:</pre><pre>cd D://biocase</pre> Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Ci dessous un exemple sous DOs pour Windows, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python donnée dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de interpréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecture depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur et permettre au modne extérieur de voir vos fichiers de configuration (qui incluent des mots de passe). Ainsi si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre server est ''d:\biocase\www' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache. Si vous installer BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez en informer l'administrateur avant de modifier ce fichier! [[image:instHttpdConf.png|480px]] Sauvez le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utlisez IIS vous devez créer d'abord un répertoire virtuel pour le répertoire physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'), utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier .cgi and .py avec le chemin de votre fichier exécutable Python. Vous devriez aussi aussi ajouter ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmons de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'accueil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du logiciel utilisé pour la base de données à publier et des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'accueil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes), qui indique également si cette librairie est présente sur le système et, si ou, dans quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci: [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe proprement dite sont déjà inclues dans le fichier d'archive que vous avez installé ou bien le répertoire Subversion d'où vous êtes partis. La librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos don"nes sur le Web) – ainsi si vous n'envisagez pas de l'utiliser il n'est pas nécessaire de l'installer. Sinon cliquez sur le lien 'Download', trouvez le fichier d'installation qui convient à votre système d'exploitation et installez 4SuiteXML à partir du fichier téléchargé. Néanmoins, Pour chaque type de base de données (SGBDR ou Système Gestionnaire be base de données Relationnelle dans le jargon), vous devrez installer le package ou driver Python correspondant. Ce driver permet à un programme Python de se connecter à une base de donénes . Il doit être installé et configuré non pas au niveau de BioCASe, mais à celui de l'environnement Python; mais néanmoins le BPS peut vous guider dans leur installation. Si votre collection est par exemple stockée dans une base de données MySQL, cliquez sur le lien pour télécharger le module MySQL, trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraires dans votre navigateur. La ligne de la liste qui correspond au driver devrait passer de l'orange pour 'non installé' à une indication verte mentionnant la version de la libraire. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utilisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessire d'installer Graphviz. Dans le cas contraire nous vous recommandons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquat, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les pramètres du BPS. A la fin de la procédure de configuration, vous devriez changer le mot de passe. Retournez à l'écran d'accueil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la magde Configuration dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation (nosu supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''). A la quatrième ligne, vous pouvez encoder un nouveau mot de passe. Soyez précautionneux en le tapant (si jamais vous l'oubliez, il est toujours possible de le réintialiser en modifiant directement le fichier de configuration). Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers binaires, car Graphviz n'est pas une librairie Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sut votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary</small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''), sous Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connection à une base de donénes), vous devriez vous assurer que tout est installé correctement: # Taper ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à votre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modificables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les Logs (fichiers qui enregistrent les opérations sur la machine) et la configuration. Sinon, le chemin vers le fichier posent problème sera montré –. Veuillez vous assurer que Python et.ou Apache peuvent écrire sur ce fichier (par exemple sur Ubuntu où Apache correspond à un compte utlsiateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre>). Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connexion via la librairie ODBC) et que vous ne souhaitez pas utiliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] 05ee4ed2055f71077d27c7b1489a78aea7d97fe1 524 523 2012-04-10T15:41:01Z FranckTheeten 15 /* Exécution du script d'installation */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes en fonction de votre degré de familiarité avec Python et Apache. Si vous êtes un débutant, veuillez lire le Guide du Débutant [[BeginnersGuide|Beginner’s Guide]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit soit être à l'extérieur d'un pare-feu institutionnel ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de satisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine à la base de données à publier - soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL Server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fonctionner également avec les versions 2.6 et 2.7, la version recommandée est de 2.5 Python (en fait la dernière sous-version stable de la branche 2.5: 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le bib site Web (par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous faites l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier Program Files par défaut. En effet à cet endroit le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir du Subversion du BGBM (Subversion ou SVN est un logiciel qui permet à plusieurs programmeurs de travailler en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la documentation des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (pré-installé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS P Oar exemple si votre provider est installé à l'emplacement ''D:\\biocase'' tapez <pre>D:</pre><pre>cd D://biocase</pre> Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Ci dessous un exemple sous DOs pour Windows, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python figurant dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de interpréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecture depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur et permettre au modne extérieur de voir vos fichiers de configuration (qui incluent des mots de passe). Ainsi si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre server est ''d:\biocase\www' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache. Si vous installer BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez en informer l'administrateur avant de modifier ce fichier! [[image:instHttpdConf.png|480px]] Sauvez le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utlisez IIS vous devez créer d'abord un répertoire virtuel pour le répertoire physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'), utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier .cgi and .py avec le chemin de votre fichier exécutable Python. Vous devriez aussi aussi ajouter ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmons de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'accueil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du logiciel utilisé pour la base de données à publier et des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'accueil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes), qui indique également si cette librairie est présente sur le système et, si ou, dans quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci: [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe proprement dite sont déjà inclues dans le fichier d'archive que vous avez installé ou bien le répertoire Subversion d'où vous êtes partis. La librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos don"nes sur le Web) – ainsi si vous n'envisagez pas de l'utiliser il n'est pas nécessaire de l'installer. Sinon cliquez sur le lien 'Download', trouvez le fichier d'installation qui convient à votre système d'exploitation et installez 4SuiteXML à partir du fichier téléchargé. Néanmoins, Pour chaque type de base de données (SGBDR ou Système Gestionnaire be base de données Relationnelle dans le jargon), vous devrez installer le package ou driver Python correspondant. Ce driver permet à un programme Python de se connecter à une base de donénes . Il doit être installé et configuré non pas au niveau de BioCASe, mais à celui de l'environnement Python; mais néanmoins le BPS peut vous guider dans leur installation. Si votre collection est par exemple stockée dans une base de données MySQL, cliquez sur le lien pour télécharger le module MySQL, trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraires dans votre navigateur. La ligne de la liste qui correspond au driver devrait passer de l'orange pour 'non installé' à une indication verte mentionnant la version de la libraire. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utilisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessire d'installer Graphviz. Dans le cas contraire nous vous recommandons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquat, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les pramètres du BPS. A la fin de la procédure de configuration, vous devriez changer le mot de passe. Retournez à l'écran d'accueil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la magde Configuration dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation (nosu supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''). A la quatrième ligne, vous pouvez encoder un nouveau mot de passe. Soyez précautionneux en le tapant (si jamais vous l'oubliez, il est toujours possible de le réintialiser en modifiant directement le fichier de configuration). Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers binaires, car Graphviz n'est pas une librairie Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sut votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary</small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''), sous Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connection à une base de donénes), vous devriez vous assurer que tout est installé correctement: # Taper ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à votre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modificables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les Logs (fichiers qui enregistrent les opérations sur la machine) et la configuration. Sinon, le chemin vers le fichier posent problème sera montré –. Veuillez vous assurer que Python et.ou Apache peuvent écrire sur ce fichier (par exemple sur Ubuntu où Apache correspond à un compte utlsiateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre>). Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connexion via la librairie ODBC) et que vous ne souhaitez pas utiliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] f5a49598bc98245e299809a2db51fb49c8a5ac15 525 524 2012-04-10T15:41:30Z FranckTheeten 15 /* Vérifiez votre Installation */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes en fonction de votre degré de familiarité avec Python et Apache. Si vous êtes un débutant, veuillez lire le Guide du Débutant [[BeginnersGuide|Beginner’s Guide]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit soit être à l'extérieur d'un pare-feu institutionnel ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de satisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine à la base de données à publier - soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL Server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fonctionner également avec les versions 2.6 et 2.7, la version recommandée est de 2.5 Python (en fait la dernière sous-version stable de la branche 2.5: 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le bib site Web (par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous faites l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier Program Files par défaut. En effet à cet endroit le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir du Subversion du BGBM (Subversion ou SVN est un logiciel qui permet à plusieurs programmeurs de travailler en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la documentation des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (pré-installé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS P Oar exemple si votre provider est installé à l'emplacement ''D:\\biocase'' tapez <pre>D:</pre><pre>cd D://biocase</pre> Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Ci dessous un exemple sous DOs pour Windows, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python figurant dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de interpréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecture depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur et permettre au modne extérieur de voir vos fichiers de configuration (qui incluent des mots de passe). Ainsi si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre server est ''d:\biocase\www' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache. Si vous installer BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez en informer l'administrateur avant de modifier ce fichier! [[image:instHttpdConf.png|480px]] Sauvez le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utlisez IIS vous devez créer d'abord un répertoire virtuel pour le répertoire physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'), utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier .cgi and .py avec le chemin de votre fichier exécutable Python. Vous devriez aussi aussi ajouter ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmons de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'accueil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du logiciel utilisé pour la base de données à publier et des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'accueil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes), qui indique également si cette librairie est présente sur le système et, si ou, dans quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci: [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe proprement dite sont déjà inclues dans le fichier d'archive que vous avez installé ou bien le répertoire Subversion d'où vous êtes partis. La librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos don"nes sur le Web) – ainsi si vous n'envisagez pas de l'utiliser il n'est pas nécessaire de l'installer. Sinon cliquez sur le lien 'Download', trouvez le fichier d'installation qui convient à votre système d'exploitation et installez 4SuiteXML à partir du fichier téléchargé. Néanmoins, Pour chaque type de base de données (SGBDR ou Système Gestionnaire be base de données Relationnelle dans le jargon), vous devrez installer le package ou driver Python correspondant. Ce driver permet à un programme Python de se connecter à une base de donénes . Il doit être installé et configuré non pas au niveau de BioCASe, mais à celui de l'environnement Python; mais néanmoins le BPS peut vous guider dans leur installation. Si votre collection est par exemple stockée dans une base de données MySQL, cliquez sur le lien pour télécharger le module MySQL, trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraires dans votre navigateur. La ligne de la liste qui correspond au driver devrait passer de l'orange pour 'non installé' à une indication verte mentionnant la version de la libraire. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utilisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessire d'installer Graphviz. Dans le cas contraire nous vous recommandons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquat, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les pramètres du BPS. A la fin de la procédure de configuration, vous devriez changer le mot de passe. Retournez à l'écran d'accueil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la magde Configuration dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation (nosu supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''). A la quatrième ligne, vous pouvez encoder un nouveau mot de passe. Soyez précautionneux en le tapant (si jamais vous l'oubliez, il est toujours possible de le réintialiser en modifiant directement le fichier de configuration). Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers binaires, car Graphviz n'est pas une librairie Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sut votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary</small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''), sous Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connexion à une base de données), vous devriez vous assurer que tout est installé correctement: # Taper ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à votre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modifiables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les 'Logs' (fichiers qui enregistrent les opérations sur la machine) et la configuration. Sinon, le chemin vers le fichier posent problème sera montré –. Veuillez vous assurer que Python et.ou Apache peuvent écrire sur ce fichier (par exemple sur Ubuntu où Apache correspond à un compte utilisateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre>). Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connexion via la librairie ODBC) et que vous ne souhaitez pas utiliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] f9c05088ba28df4cf412fdb302005de40543a37d 526 525 2012-04-10T15:42:36Z FranckTheeten 15 /* Vérifiez votre Installation */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes en fonction de votre degré de familiarité avec Python et Apache. Si vous êtes un débutant, veuillez lire le Guide du Débutant [[BeginnersGuide|Beginner’s Guide]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit soit être à l'extérieur d'un pare-feu institutionnel ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de satisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine à la base de données à publier - soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL Server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fonctionner également avec les versions 2.6 et 2.7, la version recommandée est de 2.5 Python (en fait la dernière sous-version stable de la branche 2.5: 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le bib site Web (par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous faites l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier Program Files par défaut. En effet à cet endroit le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir du Subversion du BGBM (Subversion ou SVN est un logiciel qui permet à plusieurs programmeurs de travailler en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la documentation des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (pré-installé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS P Oar exemple si votre provider est installé à l'emplacement ''D:\\biocase'' tapez <pre>D:</pre><pre>cd D://biocase</pre> Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Ci dessous un exemple sous DOs pour Windows, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python figurant dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de interpréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecture depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur et permettre au modne extérieur de voir vos fichiers de configuration (qui incluent des mots de passe). Ainsi si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre server est ''d:\biocase\www' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache. Si vous installer BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez en informer l'administrateur avant de modifier ce fichier! [[image:instHttpdConf.png|480px]] Sauvez le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utlisez IIS vous devez créer d'abord un répertoire virtuel pour le répertoire physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'), utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier .cgi and .py avec le chemin de votre fichier exécutable Python. Vous devriez aussi aussi ajouter ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmons de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'accueil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du logiciel utilisé pour la base de données à publier et des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'accueil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes), qui indique également si cette librairie est présente sur le système et, si ou, dans quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci: [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe proprement dite sont déjà inclues dans le fichier d'archive que vous avez installé ou bien le répertoire Subversion d'où vous êtes partis. La librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos don"nes sur le Web) – ainsi si vous n'envisagez pas de l'utiliser il n'est pas nécessaire de l'installer. Sinon cliquez sur le lien 'Download', trouvez le fichier d'installation qui convient à votre système d'exploitation et installez 4SuiteXML à partir du fichier téléchargé. Néanmoins, Pour chaque type de base de données (SGBDR ou Système Gestionnaire be base de données Relationnelle dans le jargon), vous devrez installer le package ou driver Python correspondant. Ce driver permet à un programme Python de se connecter à une base de donénes . Il doit être installé et configuré non pas au niveau de BioCASe, mais à celui de l'environnement Python; mais néanmoins le BPS peut vous guider dans leur installation. Si votre collection est par exemple stockée dans une base de données MySQL, cliquez sur le lien pour télécharger le module MySQL, trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraires dans votre navigateur. La ligne de la liste qui correspond au driver devrait passer de l'orange pour 'non installé' à une indication verte mentionnant la version de la libraire. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utilisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessire d'installer Graphviz. Dans le cas contraire nous vous recommandons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquat, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les pramètres du BPS. A la fin de la procédure de configuration, vous devriez changer le mot de passe. Retournez à l'écran d'accueil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la magde Configuration dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation (nosu supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''). A la quatrième ligne, vous pouvez encoder un nouveau mot de passe. Soyez précautionneux en le tapant (si jamais vous l'oubliez, il est toujours possible de le réintialiser en modifiant directement le fichier de configuration). Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers binaires, car Graphviz n'est pas une librairie Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sut votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary</small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''), sous Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connexion à une base de données), vous devriez vous assurer que tout est installé correctement: # Tapez ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à votre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modifiables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les 'Logs' (fichiers qui enregistrent les opérations sur la machine) et la configuration. Si ce n'est pas le cas, le chemin vers le fichier posent problème sera montré. Veuillez vous assurer que Python et/ou Apache peuvent écrire sur ce fichier. Par par exemple sur Ubuntu où Apache correspond à un compte utilisateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre> Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connexion via la librairie ODBC) et que vous ne souhaitez pas utiliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] 931010c2b88343040fa389b118a5ee2605a96e34 527 526 2012-04-10T15:43:09Z FranckTheeten 15 /* Vérifiez votre Installation */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes en fonction de votre degré de familiarité avec Python et Apache. Si vous êtes un débutant, veuillez lire le Guide du Débutant [[BeginnersGuide|Beginner’s Guide]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit soit être à l'extérieur d'un pare-feu institutionnel ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de satisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine à la base de données à publier - soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL Server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fonctionner également avec les versions 2.6 et 2.7, la version recommandée est de 2.5 Python (en fait la dernière sous-version stable de la branche 2.5: 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le bib site Web (par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous faites l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier Program Files par défaut. En effet à cet endroit le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir du Subversion du BGBM (Subversion ou SVN est un logiciel qui permet à plusieurs programmeurs de travailler en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la documentation des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (pré-installé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS P Oar exemple si votre provider est installé à l'emplacement ''D:\\biocase'' tapez <pre>D:</pre><pre>cd D://biocase</pre> Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Ci dessous un exemple sous DOs pour Windows, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python figurant dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de interpréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecture depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur et permettre au modne extérieur de voir vos fichiers de configuration (qui incluent des mots de passe). Ainsi si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre server est ''d:\biocase\www' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache. Si vous installer BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez en informer l'administrateur avant de modifier ce fichier! [[image:instHttpdConf.png|480px]] Sauvez le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utlisez IIS vous devez créer d'abord un répertoire virtuel pour le répertoire physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'), utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier .cgi and .py avec le chemin de votre fichier exécutable Python. Vous devriez aussi aussi ajouter ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmons de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'accueil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du logiciel utilisé pour la base de données à publier et des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'accueil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes), qui indique également si cette librairie est présente sur le système et, si ou, dans quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci: [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe proprement dite sont déjà inclues dans le fichier d'archive que vous avez installé ou bien le répertoire Subversion d'où vous êtes partis. La librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos don"nes sur le Web) – ainsi si vous n'envisagez pas de l'utiliser il n'est pas nécessaire de l'installer. Sinon cliquez sur le lien 'Download', trouvez le fichier d'installation qui convient à votre système d'exploitation et installez 4SuiteXML à partir du fichier téléchargé. Néanmoins, Pour chaque type de base de données (SGBDR ou Système Gestionnaire be base de données Relationnelle dans le jargon), vous devrez installer le package ou driver Python correspondant. Ce driver permet à un programme Python de se connecter à une base de donénes . Il doit être installé et configuré non pas au niveau de BioCASe, mais à celui de l'environnement Python; mais néanmoins le BPS peut vous guider dans leur installation. Si votre collection est par exemple stockée dans une base de données MySQL, cliquez sur le lien pour télécharger le module MySQL, trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraires dans votre navigateur. La ligne de la liste qui correspond au driver devrait passer de l'orange pour 'non installé' à une indication verte mentionnant la version de la libraire. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utilisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessire d'installer Graphviz. Dans le cas contraire nous vous recommandons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquat, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les pramètres du BPS. A la fin de la procédure de configuration, vous devriez changer le mot de passe. Retournez à l'écran d'accueil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la magde Configuration dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation (nosu supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''). A la quatrième ligne, vous pouvez encoder un nouveau mot de passe. Soyez précautionneux en le tapant (si jamais vous l'oubliez, il est toujours possible de le réintialiser en modifiant directement le fichier de configuration). Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers binaires, car Graphviz n'est pas une librairie Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sut votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary</small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''), sous Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connexion à une base de données), vous devriez vous assurer que tout est installé correctement: # Tapez ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à votre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modifiables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les 'Logs' (fichiers qui enregistrent les opérations sur la machine) et la configuration. Si ce n'est pas le cas, le chemin vers le fichier qui pose problème sera montré. Veuillez vous assurer que Python et/ou Apache peuvent écrire sur ce fichier. Par par exemple sur Ubuntu où Apache correspond à un compte utilisateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre> Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connexion via la librairie ODBC) et que vous ne souhaitez pas utiliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] 296e0d7101a936f021839ae9b080a0e14d4c829f 528 527 2012-04-10T15:45:30Z FranckTheeten 15 /* Changer le mot de passe par défaut */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes en fonction de votre degré de familiarité avec Python et Apache. Si vous êtes un débutant, veuillez lire le Guide du Débutant [[BeginnersGuide|Beginner’s Guide]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit soit être à l'extérieur d'un pare-feu institutionnel ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de satisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine à la base de données à publier - soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL Server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fonctionner également avec les versions 2.6 et 2.7, la version recommandée est de 2.5 Python (en fait la dernière sous-version stable de la branche 2.5: 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le bib site Web (par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous faites l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier Program Files par défaut. En effet à cet endroit le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir du Subversion du BGBM (Subversion ou SVN est un logiciel qui permet à plusieurs programmeurs de travailler en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la documentation des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (pré-installé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS P Oar exemple si votre provider est installé à l'emplacement ''D:\\biocase'' tapez <pre>D:</pre><pre>cd D://biocase</pre> Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Ci dessous un exemple sous DOs pour Windows, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python figurant dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de interpréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecture depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur et permettre au modne extérieur de voir vos fichiers de configuration (qui incluent des mots de passe). Ainsi si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre server est ''d:\biocase\www' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache. Si vous installer BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez en informer l'administrateur avant de modifier ce fichier! [[image:instHttpdConf.png|480px]] Sauvez le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utlisez IIS vous devez créer d'abord un répertoire virtuel pour le répertoire physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'), utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier .cgi and .py avec le chemin de votre fichier exécutable Python. Vous devriez aussi aussi ajouter ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmons de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'accueil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du logiciel utilisé pour la base de données à publier et des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'accueil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes), qui indique également si cette librairie est présente sur le système et, si ou, dans quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci: [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe proprement dite sont déjà inclues dans le fichier d'archive que vous avez installé ou bien le répertoire Subversion d'où vous êtes partis. La librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos don"nes sur le Web) – ainsi si vous n'envisagez pas de l'utiliser il n'est pas nécessaire de l'installer. Sinon cliquez sur le lien 'Download', trouvez le fichier d'installation qui convient à votre système d'exploitation et installez 4SuiteXML à partir du fichier téléchargé. Néanmoins, Pour chaque type de base de données (SGBDR ou Système Gestionnaire be base de données Relationnelle dans le jargon), vous devrez installer le package ou driver Python correspondant. Ce driver permet à un programme Python de se connecter à une base de donénes . Il doit être installé et configuré non pas au niveau de BioCASe, mais à celui de l'environnement Python; mais néanmoins le BPS peut vous guider dans leur installation. Si votre collection est par exemple stockée dans une base de données MySQL, cliquez sur le lien pour télécharger le module MySQL, trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraires dans votre navigateur. La ligne de la liste qui correspond au driver devrait passer de l'orange pour 'non installé' à une indication verte mentionnant la version de la libraire. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utilisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessire d'installer Graphviz. Dans le cas contraire nous vous recommandons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquat, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == A la fin de la procédure de configuration, vous devriez changer le mot de passe. L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les paramètres du BPS, dont le mot de passe. Retournez à l'écran d'accueil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la page 'Configuration' dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation. Nous supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''. Vous pouvez encoder un nouveau mot de passe à la quatrièrme ligne de cet écran. Soyez précautionneux en le tapant. Si jamais vous l'oubliez, il est toutefois possible de le réinitialiser en modifiant directement les fichiers de configuration. Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers binaires, car Graphviz n'est pas une librairie Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sur votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary</small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''). Sur un système Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connexion à une base de données), vous devriez vous assurer que tout est installé correctement: # Tapez ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à votre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modifiables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les 'Logs' (fichiers qui enregistrent les opérations sur la machine) et la configuration. Si ce n'est pas le cas, le chemin vers le fichier qui pose problème sera montré. Veuillez vous assurer que Python et/ou Apache peuvent écrire sur ce fichier. Par par exemple sur Ubuntu où Apache correspond à un compte utilisateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre> Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connexion via la librairie ODBC) et que vous ne souhaitez pas utiliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] cc8481241743f0c631032dd207532f22b7bd38b7 529 528 2012-04-10T15:47:01Z FranckTheeten 15 /* Installer les paquets additionnels Python */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes en fonction de votre degré de familiarité avec Python et Apache. Si vous êtes un débutant, veuillez lire le Guide du Débutant [[BeginnersGuide|Beginner’s Guide]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit soit être à l'extérieur d'un pare-feu institutionnel ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de satisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine à la base de données à publier - soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL Server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fonctionner également avec les versions 2.6 et 2.7, la version recommandée est de 2.5 Python (en fait la dernière sous-version stable de la branche 2.5: 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le bib site Web (par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous faites l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier Program Files par défaut. En effet à cet endroit le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir du Subversion du BGBM (Subversion ou SVN est un logiciel qui permet à plusieurs programmeurs de travailler en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la documentation des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (pré-installé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS P Oar exemple si votre provider est installé à l'emplacement ''D:\\biocase'' tapez <pre>D:</pre><pre>cd D://biocase</pre> Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Ci dessous un exemple sous DOs pour Windows, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python figurant dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de interpréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecture depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur et permettre au modne extérieur de voir vos fichiers de configuration (qui incluent des mots de passe). Ainsi si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre server est ''d:\biocase\www' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache. Si vous installer BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez en informer l'administrateur avant de modifier ce fichier! [[image:instHttpdConf.png|480px]] Sauvez le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utlisez IIS vous devez créer d'abord un répertoire virtuel pour le répertoire physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'), utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier .cgi and .py avec le chemin de votre fichier exécutable Python. Vous devriez aussi aussi ajouter ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmons de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'accueil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du type de logiciel de la base de données à publier ainsi que des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'accueil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes). Cette page indique également si cette librairie est présente sur le système et, si dans ce cas, avec quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci: [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe proprement dite sont déjà inclues dans le fichier d'archive que vous avez installé ou bien le répertoire Subversion d'où vous êtes partis. La librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos don"nes sur le Web) – ainsi si vous n'envisagez pas de l'utiliser il n'est pas nécessaire de l'installer. Sinon cliquez sur le lien 'Download', trouvez le fichier d'installation qui convient à votre système d'exploitation et installez 4SuiteXML à partir du fichier téléchargé. Néanmoins, Pour chaque type de base de données (SGBDR ou Système Gestionnaire be base de données Relationnelle dans le jargon), vous devrez installer le package ou driver Python correspondant. Ce driver permet à un programme Python de se connecter à une base de donénes . Il doit être installé et configuré non pas au niveau de BioCASe, mais à celui de l'environnement Python; mais néanmoins le BPS peut vous guider dans leur installation. Si votre collection est par exemple stockée dans une base de données MySQL, cliquez sur le lien pour télécharger le module MySQL, trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraires dans votre navigateur. La ligne de la liste qui correspond au driver devrait passer de l'orange pour 'non installé' à une indication verte mentionnant la version de la libraire. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utilisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessire d'installer Graphviz. Dans le cas contraire nous vous recommandons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquat, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == A la fin de la procédure de configuration, vous devriez changer le mot de passe. L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les paramètres du BPS, dont le mot de passe. Retournez à l'écran d'accueil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la page 'Configuration' dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation. Nous supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''. Vous pouvez encoder un nouveau mot de passe à la quatrièrme ligne de cet écran. Soyez précautionneux en le tapant. Si jamais vous l'oubliez, il est toutefois possible de le réinitialiser en modifiant directement les fichiers de configuration. Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers binaires, car Graphviz n'est pas une librairie Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sur votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary</small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''). Sur un système Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connexion à une base de données), vous devriez vous assurer que tout est installé correctement: # Tapez ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à votre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modifiables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les 'Logs' (fichiers qui enregistrent les opérations sur la machine) et la configuration. Si ce n'est pas le cas, le chemin vers le fichier qui pose problème sera montré. Veuillez vous assurer que Python et/ou Apache peuvent écrire sur ce fichier. Par par exemple sur Ubuntu où Apache correspond à un compte utilisateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre> Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connexion via la librairie ODBC) et que vous ne souhaitez pas utiliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] eea83dfddadf9a949646fb3d9cbe18820540de8d 530 529 2012-04-10T15:49:04Z FranckTheeten 15 /* Installer les paquets additionnels Python */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes en fonction de votre degré de familiarité avec Python et Apache. Si vous êtes un débutant, veuillez lire le Guide du Débutant [[BeginnersGuide|Beginner’s Guide]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit soit être à l'extérieur d'un pare-feu institutionnel ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de satisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine à la base de données à publier - soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL Server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fonctionner également avec les versions 2.6 et 2.7, la version recommandée est de 2.5 Python (en fait la dernière sous-version stable de la branche 2.5: 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le bib site Web (par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous faites l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier Program Files par défaut. En effet à cet endroit le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir du Subversion du BGBM (Subversion ou SVN est un logiciel qui permet à plusieurs programmeurs de travailler en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la documentation des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (pré-installé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS P Oar exemple si votre provider est installé à l'emplacement ''D:\\biocase'' tapez <pre>D:</pre><pre>cd D://biocase</pre> Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Ci dessous un exemple sous DOs pour Windows, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python figurant dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de interpréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecture depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur et permettre au modne extérieur de voir vos fichiers de configuration (qui incluent des mots de passe). Ainsi si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre server est ''d:\biocase\www' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache. Si vous installer BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez en informer l'administrateur avant de modifier ce fichier! [[image:instHttpdConf.png|480px]] Sauvez le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utlisez IIS vous devez créer d'abord un répertoire virtuel pour le répertoire physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'), utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier .cgi and .py avec le chemin de votre fichier exécutable Python. Vous devriez aussi aussi ajouter ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmons de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'accueil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du type de logiciel de la base de données à publier ainsi que des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'accueil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes). Cette page indique également si cette librairie est présente sur le système et, si dans ce cas, avec quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci: [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe proprement dites sont déjà inclues dans le fichier d'archive que vous avez installé ou bien dans le répertoire Subversion d'où vous êtes parti. La librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos donnes sur le Web). Si vous n'envisagez pas de l'utiliser, il n'est pas nécessaire de l'installer. Mais sinon cliquez sur le lien 'Download', trouvez le fichier d'installation qui convient à votre système d'exploitation, et installez 4SuiteXML à partir du fichier téléchargé. Pour chaque type de base de données (SGBDR ou Système Gestionnaire be base de données Relationnelle dans le jargon), vous devrez toujours installer le package ou driver Python correspondant. Ce driver permet à un programme Python de se connecter à une base de données. Il doit être installé et configuré non pas au niveau de BioCASe, mais au niveau de l'environnement Python. Le BPS peut toutefois vous guider dans leur installation. Si votre collection est par exemple stockée dans une base de données MySQL, cliquez sur le lien pour télécharger le module MySQL, trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraires dans votre navigateur. La ligne de la liste qui correspond au driver devrait passer de l'orange pour 'non installé' à une indication verte mentionnant la version de la libraire. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utilisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessire d'installer Graphviz. Dans le cas contraire nous vous recommandons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquat, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == A la fin de la procédure de configuration, vous devriez changer le mot de passe. L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les paramètres du BPS, dont le mot de passe. Retournez à l'écran d'accueil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la page 'Configuration' dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation. Nous supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''. Vous pouvez encoder un nouveau mot de passe à la quatrièrme ligne de cet écran. Soyez précautionneux en le tapant. Si jamais vous l'oubliez, il est toutefois possible de le réinitialiser en modifiant directement les fichiers de configuration. Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers binaires, car Graphviz n'est pas une librairie Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sur votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary</small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''). Sur un système Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connexion à une base de données), vous devriez vous assurer que tout est installé correctement: # Tapez ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à votre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modifiables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les 'Logs' (fichiers qui enregistrent les opérations sur la machine) et la configuration. Si ce n'est pas le cas, le chemin vers le fichier qui pose problème sera montré. Veuillez vous assurer que Python et/ou Apache peuvent écrire sur ce fichier. Par par exemple sur Ubuntu où Apache correspond à un compte utilisateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre> Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connexion via la librairie ODBC) et que vous ne souhaitez pas utiliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] f6fb33215a302ded5d08a3e4519093df4ed5c988 531 530 2012-04-10T15:49:42Z FranckTheeten 15 /* Installer les paquets additionnels Python */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes en fonction de votre degré de familiarité avec Python et Apache. Si vous êtes un débutant, veuillez lire le Guide du Débutant [[BeginnersGuide|Beginner’s Guide]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit soit être à l'extérieur d'un pare-feu institutionnel ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de satisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine à la base de données à publier - soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL Server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fonctionner également avec les versions 2.6 et 2.7, la version recommandée est de 2.5 Python (en fait la dernière sous-version stable de la branche 2.5: 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le bib site Web (par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous faites l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier Program Files par défaut. En effet à cet endroit le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir du Subversion du BGBM (Subversion ou SVN est un logiciel qui permet à plusieurs programmeurs de travailler en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la documentation des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (pré-installé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS P Oar exemple si votre provider est installé à l'emplacement ''D:\\biocase'' tapez <pre>D:</pre><pre>cd D://biocase</pre> Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Ci dessous un exemple sous DOs pour Windows, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python figurant dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de interpréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecture depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur et permettre au modne extérieur de voir vos fichiers de configuration (qui incluent des mots de passe). Ainsi si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre server est ''d:\biocase\www' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache. Si vous installer BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez en informer l'administrateur avant de modifier ce fichier! [[image:instHttpdConf.png|480px]] Sauvez le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utlisez IIS vous devez créer d'abord un répertoire virtuel pour le répertoire physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'), utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier .cgi and .py avec le chemin de votre fichier exécutable Python. Vous devriez aussi aussi ajouter ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmons de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'accueil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du type de logiciel de la base de données à publier ainsi que des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'accueil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes). Cette page indique également si cette librairie est présente sur le système et, si dans ce cas, avec quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci: [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe proprement dites sont déjà inclues dans le fichier d'archive que vous avez installé ou bien dans le répertoire Subversion d'où vous êtes parti. La librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos donnes sur le Web). Si vous n'envisagez pas de l'utiliser, il n'est pas nécessaire de l'installer. Mais sinon cliquez sur le lien 'Download', trouvez le fichier d'installation qui convient à votre système d'exploitation, et installez 4SuiteXML à partir du fichier téléchargé. Pour chaque type de base de données (SGBDR ou Système Gestionnaire be base de données Relationnelle dans le jargon), vous devrez toujours installer le package ou driver Python correspondant. Ce driver permet à un programme Python de se connecter à une base de données. Il doit être installé et configuré non pas au niveau de BioCASe, mais au niveau de l'environnement Python. Le BPS peut toutefois vous guider dans leur installation. Si votre collection est par exemple stockée dans une base de données 'MySQL', cliquez sur le lien pour télécharger le module 'MySQL', trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraires dans votre navigateur. La ligne de la liste qui correspond au driver devrait passer de l'orange pour 'non installé' à une indication verte mentionnant la version de la libraire. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utilisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessire d'installer Graphviz. Dans le cas contraire nous vous recommandons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquat, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == A la fin de la procédure de configuration, vous devriez changer le mot de passe. L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les paramètres du BPS, dont le mot de passe. Retournez à l'écran d'accueil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la page 'Configuration' dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation. Nous supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''. Vous pouvez encoder un nouveau mot de passe à la quatrièrme ligne de cet écran. Soyez précautionneux en le tapant. Si jamais vous l'oubliez, il est toutefois possible de le réinitialiser en modifiant directement les fichiers de configuration. Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers binaires, car Graphviz n'est pas une librairie Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sur votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary</small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''). Sur un système Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connexion à une base de données), vous devriez vous assurer que tout est installé correctement: # Tapez ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à votre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modifiables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les 'Logs' (fichiers qui enregistrent les opérations sur la machine) et la configuration. Si ce n'est pas le cas, le chemin vers le fichier qui pose problème sera montré. Veuillez vous assurer que Python et/ou Apache peuvent écrire sur ce fichier. Par par exemple sur Ubuntu où Apache correspond à un compte utilisateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre> Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connexion via la librairie ODBC) et que vous ne souhaitez pas utiliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] 039c63e47629a64cb26d493e053e440fbae16e6a 532 531 2012-04-10T15:50:09Z FranckTheeten 15 /* Installer les paquets additionnels Python */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes en fonction de votre degré de familiarité avec Python et Apache. Si vous êtes un débutant, veuillez lire le Guide du Débutant [[BeginnersGuide|Beginner’s Guide]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit soit être à l'extérieur d'un pare-feu institutionnel ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de satisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine à la base de données à publier - soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL Server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fonctionner également avec les versions 2.6 et 2.7, la version recommandée est de 2.5 Python (en fait la dernière sous-version stable de la branche 2.5: 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le bib site Web (par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous faites l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier Program Files par défaut. En effet à cet endroit le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir du Subversion du BGBM (Subversion ou SVN est un logiciel qui permet à plusieurs programmeurs de travailler en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la documentation des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (pré-installé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS P Oar exemple si votre provider est installé à l'emplacement ''D:\\biocase'' tapez <pre>D:</pre><pre>cd D://biocase</pre> Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Ci dessous un exemple sous DOs pour Windows, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python figurant dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de interpréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecture depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur et permettre au modne extérieur de voir vos fichiers de configuration (qui incluent des mots de passe). Ainsi si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre server est ''d:\biocase\www' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache. Si vous installer BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez en informer l'administrateur avant de modifier ce fichier! [[image:instHttpdConf.png|480px]] Sauvez le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utlisez IIS vous devez créer d'abord un répertoire virtuel pour le répertoire physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'), utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier .cgi and .py avec le chemin de votre fichier exécutable Python. Vous devriez aussi aussi ajouter ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmons de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'accueil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du type de logiciel de la base de données à publier ainsi que des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'accueil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes). Cette page indique également si cette librairie est présente sur le système et, si dans ce cas, avec quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci: [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe proprement dites sont déjà inclues dans le fichier d'archive que vous avez installé ou bien dans le répertoire Subversion d'où vous êtes parti. La librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos donnes sur le Web). Si vous n'envisagez pas de l'utiliser, il n'est pas nécessaire de l'installer. Mais sinon cliquez sur le lien 'Download', trouvez le fichier d'installation qui convient à votre système d'exploitation, et installez 4SuiteXML à partir du fichier téléchargé. Pour chaque type de base de données (SGBDR ou Système Gestionnaire be base de données Relationnelle dans le jargon), vous devrez toujours installer le package ou driver Python correspondant. Ce driver permet à un programme Python de se connecter à une base de données. Il doit être installé et configuré non pas au niveau de BioCASe, mais au niveau de l'environnement Python. Le BPS peut toutefois vous guider dans leur installation. Si votre collection est par exemple stockée dans une base de données 'MySQL', cliquez sur le lien pour télécharger le module 'MySQL', trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraires dans votre navigateur. La ligne de la liste qui correspond au driver devrait passer de l'orange pour 'non installé' à une indication verte mentionnant la version de la libraire. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utilisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]], actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessaire d'installer Graphviz. Dans le cas contraire nous vous recommandons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquat, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == A la fin de la procédure de configuration, vous devriez changer le mot de passe. L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les paramètres du BPS, dont le mot de passe. Retournez à l'écran d'accueil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la page 'Configuration' dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation. Nous supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''. Vous pouvez encoder un nouveau mot de passe à la quatrièrme ligne de cet écran. Soyez précautionneux en le tapant. Si jamais vous l'oubliez, il est toutefois possible de le réinitialiser en modifiant directement les fichiers de configuration. Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers binaires, car Graphviz n'est pas une librairie Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sur votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary</small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''). Sur un système Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connexion à une base de données), vous devriez vous assurer que tout est installé correctement: # Tapez ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à votre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modifiables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les 'Logs' (fichiers qui enregistrent les opérations sur la machine) et la configuration. Si ce n'est pas le cas, le chemin vers le fichier qui pose problème sera montré. Veuillez vous assurer que Python et/ou Apache peuvent écrire sur ce fichier. Par par exemple sur Ubuntu où Apache correspond à un compte utilisateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre> Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connexion via la librairie ODBC) et que vous ne souhaitez pas utiliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] 9584a82c0c903067b59f2a72374fbd12f27226e7 533 532 2012-04-10T15:51:32Z FranckTheeten 15 /* Internet Information Server (IIS) */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes en fonction de votre degré de familiarité avec Python et Apache. Si vous êtes un débutant, veuillez lire le Guide du Débutant [[BeginnersGuide|Beginner’s Guide]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit soit être à l'extérieur d'un pare-feu institutionnel ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de satisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine à la base de données à publier - soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL Server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fonctionner également avec les versions 2.6 et 2.7, la version recommandée est de 2.5 Python (en fait la dernière sous-version stable de la branche 2.5: 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le bib site Web (par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous faites l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier Program Files par défaut. En effet à cet endroit le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir du Subversion du BGBM (Subversion ou SVN est un logiciel qui permet à plusieurs programmeurs de travailler en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la documentation des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (pré-installé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS P Oar exemple si votre provider est installé à l'emplacement ''D:\\biocase'' tapez <pre>D:</pre><pre>cd D://biocase</pre> Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Ci dessous un exemple sous DOs pour Windows, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python figurant dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de interpréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecture depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur et permettre au modne extérieur de voir vos fichiers de configuration (qui incluent des mots de passe). Ainsi si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre server est ''d:\biocase\www' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache. Si vous installer BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez en informer l'administrateur avant de modifier ce fichier! [[image:instHttpdConf.png|480px]] Sauvez le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utilisez IIS, vous devez créer d'abord un répertoire virtuel pour le chemin physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'). Utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier '''.cgi''' et '''.py''' avec le chemin de votre fichier exécutable Python. Vous devriez ajouter également ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmoins de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'accueil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du type de logiciel de la base de données à publier ainsi que des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'accueil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes). Cette page indique également si cette librairie est présente sur le système et, si dans ce cas, avec quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci: [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe proprement dites sont déjà inclues dans le fichier d'archive que vous avez installé ou bien dans le répertoire Subversion d'où vous êtes parti. La librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos donnes sur le Web). Si vous n'envisagez pas de l'utiliser, il n'est pas nécessaire de l'installer. Mais sinon cliquez sur le lien 'Download', trouvez le fichier d'installation qui convient à votre système d'exploitation, et installez 4SuiteXML à partir du fichier téléchargé. Pour chaque type de base de données (SGBDR ou Système Gestionnaire be base de données Relationnelle dans le jargon), vous devrez toujours installer le package ou driver Python correspondant. Ce driver permet à un programme Python de se connecter à une base de données. Il doit être installé et configuré non pas au niveau de BioCASe, mais au niveau de l'environnement Python. Le BPS peut toutefois vous guider dans leur installation. Si votre collection est par exemple stockée dans une base de données 'MySQL', cliquez sur le lien pour télécharger le module 'MySQL', trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraires dans votre navigateur. La ligne de la liste qui correspond au driver devrait passer de l'orange pour 'non installé' à une indication verte mentionnant la version de la libraire. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utilisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]], actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessaire d'installer Graphviz. Dans le cas contraire nous vous recommandons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquat, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == A la fin de la procédure de configuration, vous devriez changer le mot de passe. L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les paramètres du BPS, dont le mot de passe. Retournez à l'écran d'accueil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la page 'Configuration' dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation. Nous supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''. Vous pouvez encoder un nouveau mot de passe à la quatrièrme ligne de cet écran. Soyez précautionneux en le tapant. Si jamais vous l'oubliez, il est toutefois possible de le réinitialiser en modifiant directement les fichiers de configuration. Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers binaires, car Graphviz n'est pas une librairie Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sur votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary</small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''). Sur un système Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connexion à une base de données), vous devriez vous assurer que tout est installé correctement: # Tapez ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à votre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modifiables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les 'Logs' (fichiers qui enregistrent les opérations sur la machine) et la configuration. Si ce n'est pas le cas, le chemin vers le fichier qui pose problème sera montré. Veuillez vous assurer que Python et/ou Apache peuvent écrire sur ce fichier. Par par exemple sur Ubuntu où Apache correspond à un compte utilisateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre> Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connexion via la librairie ODBC) et que vous ne souhaitez pas utiliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] eb455d35f3d72827b24b6b691d3321ac3e609feb 534 533 2012-04-10T15:52:50Z FranckTheeten 15 /* Configurer votre serveur Web pour BioCASe */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes en fonction de votre degré de familiarité avec Python et Apache. Si vous êtes un débutant, veuillez lire le Guide du Débutant [[BeginnersGuide|Beginner’s Guide]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit soit être à l'extérieur d'un pare-feu institutionnel ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de satisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine à la base de données à publier - soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL Server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fonctionner également avec les versions 2.6 et 2.7, la version recommandée est de 2.5 Python (en fait la dernière sous-version stable de la branche 2.5: 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le bib site Web (par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous faites l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier Program Files par défaut. En effet à cet endroit le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir du Subversion du BGBM (Subversion ou SVN est un logiciel qui permet à plusieurs programmeurs de travailler en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la documentation des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (pré-installé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS P Oar exemple si votre provider est installé à l'emplacement ''D:\\biocase'' tapez <pre>D:</pre><pre>cd D://biocase</pre> Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Ci dessous un exemple sous DOs pour Windows, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python figurant dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de interpréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecture depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur en permettant au monde extérieur de voir vos fichiers de configuration , qui incluent des mots de passe. Par exemple, si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre serveur est ''d:\biocase\www' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache. Si vous installer BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez en informer l'administrateur avant de modifier ce fichier! [[image:instHttpdConf.png|480px]] Sauvez le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utilisez IIS, vous devez créer d'abord un répertoire virtuel pour le chemin physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'). Utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier '''.cgi''' et '''.py''' avec le chemin de votre fichier exécutable Python. Vous devriez ajouter également ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmoins de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'accueil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du type de logiciel de la base de données à publier ainsi que des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'accueil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes). Cette page indique également si cette librairie est présente sur le système et, si dans ce cas, avec quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci: [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe proprement dites sont déjà inclues dans le fichier d'archive que vous avez installé ou bien dans le répertoire Subversion d'où vous êtes parti. La librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos donnes sur le Web). Si vous n'envisagez pas de l'utiliser, il n'est pas nécessaire de l'installer. Mais sinon cliquez sur le lien 'Download', trouvez le fichier d'installation qui convient à votre système d'exploitation, et installez 4SuiteXML à partir du fichier téléchargé. Pour chaque type de base de données (SGBDR ou Système Gestionnaire be base de données Relationnelle dans le jargon), vous devrez toujours installer le package ou driver Python correspondant. Ce driver permet à un programme Python de se connecter à une base de données. Il doit être installé et configuré non pas au niveau de BioCASe, mais au niveau de l'environnement Python. Le BPS peut toutefois vous guider dans leur installation. Si votre collection est par exemple stockée dans une base de données 'MySQL', cliquez sur le lien pour télécharger le module 'MySQL', trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraires dans votre navigateur. La ligne de la liste qui correspond au driver devrait passer de l'orange pour 'non installé' à une indication verte mentionnant la version de la libraire. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utilisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]], actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessaire d'installer Graphviz. Dans le cas contraire nous vous recommandons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquat, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == A la fin de la procédure de configuration, vous devriez changer le mot de passe. L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les paramètres du BPS, dont le mot de passe. Retournez à l'écran d'accueil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la page 'Configuration' dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation. Nous supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''. Vous pouvez encoder un nouveau mot de passe à la quatrièrme ligne de cet écran. Soyez précautionneux en le tapant. Si jamais vous l'oubliez, il est toutefois possible de le réinitialiser en modifiant directement les fichiers de configuration. Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers binaires, car Graphviz n'est pas une librairie Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sur votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary</small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''). Sur un système Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connexion à une base de données), vous devriez vous assurer que tout est installé correctement: # Tapez ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à votre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modifiables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les 'Logs' (fichiers qui enregistrent les opérations sur la machine) et la configuration. Si ce n'est pas le cas, le chemin vers le fichier qui pose problème sera montré. Veuillez vous assurer que Python et/ou Apache peuvent écrire sur ce fichier. Par par exemple sur Ubuntu où Apache correspond à un compte utilisateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre> Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connexion via la librairie ODBC) et que vous ne souhaitez pas utiliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] 2e29af765a57f6ec0ed7ae59c63b9c2853863a0a 535 534 2012-04-10T15:55:44Z FranckTheeten 15 /* Configurer votre serveur Web pour BioCASe */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes en fonction de votre degré de familiarité avec Python et Apache. Si vous êtes un débutant, veuillez lire le Guide du Débutant [[BeginnersGuide|Beginner’s Guide]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit soit être à l'extérieur d'un pare-feu institutionnel ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de satisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine à la base de données à publier - soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL Server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fonctionner également avec les versions 2.6 et 2.7, la version recommandée est de 2.5 Python (en fait la dernière sous-version stable de la branche 2.5: 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le bib site Web (par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous faites l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier Program Files par défaut. En effet à cet endroit le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir du Subversion du BGBM (Subversion ou SVN est un logiciel qui permet à plusieurs programmeurs de travailler en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la documentation des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (pré-installé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS P Oar exemple si votre provider est installé à l'emplacement ''D:\\biocase'' tapez <pre>D:</pre><pre>cd D://biocase</pre> Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Ci dessous un exemple sous DOs pour Windows, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python figurant dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de interpréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecture depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur en permettant au monde extérieur de voir vos fichiers de configuration , qui incluent des mots de passe. Par exemple, si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre serveur est ''d:\biocase\www'' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache. Si vous installer BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez en informer l'administrateur avant de modifier ce fichier! [[image:instHttpdConf.png|480px]] Sauvez le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utilisez IIS, vous devez créer d'abord un répertoire virtuel pour le chemin physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'). Utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier '''.cgi''' et '''.py''' avec le chemin de votre fichier exécutable Python. Vous devriez ajouter également ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmoins de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'accueil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du type de logiciel de la base de données à publier ainsi que des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'accueil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes). Cette page indique également si cette librairie est présente sur le système et, si dans ce cas, avec quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci: [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe proprement dites sont déjà inclues dans le fichier d'archive que vous avez installé ou bien dans le répertoire Subversion d'où vous êtes parti. La librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos donnes sur le Web). Si vous n'envisagez pas de l'utiliser, il n'est pas nécessaire de l'installer. Mais sinon cliquez sur le lien 'Download', trouvez le fichier d'installation qui convient à votre système d'exploitation, et installez 4SuiteXML à partir du fichier téléchargé. Pour chaque type de base de données (SGBDR ou Système Gestionnaire be base de données Relationnelle dans le jargon), vous devrez toujours installer le package ou driver Python correspondant. Ce driver permet à un programme Python de se connecter à une base de données. Il doit être installé et configuré non pas au niveau de BioCASe, mais au niveau de l'environnement Python. Le BPS peut toutefois vous guider dans leur installation. Si votre collection est par exemple stockée dans une base de données 'MySQL', cliquez sur le lien pour télécharger le module 'MySQL', trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraires dans votre navigateur. La ligne de la liste qui correspond au driver devrait passer de l'orange pour 'non installé' à une indication verte mentionnant la version de la libraire. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utilisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]], actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessaire d'installer Graphviz. Dans le cas contraire nous vous recommandons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquat, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == A la fin de la procédure de configuration, vous devriez changer le mot de passe. L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les paramètres du BPS, dont le mot de passe. Retournez à l'écran d'accueil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la page 'Configuration' dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation. Nous supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''. Vous pouvez encoder un nouveau mot de passe à la quatrièrme ligne de cet écran. Soyez précautionneux en le tapant. Si jamais vous l'oubliez, il est toutefois possible de le réinitialiser en modifiant directement les fichiers de configuration. Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers binaires, car Graphviz n'est pas une librairie Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sur votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary</small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''). Sur un système Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connexion à une base de données), vous devriez vous assurer que tout est installé correctement: # Tapez ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à votre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modifiables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les 'Logs' (fichiers qui enregistrent les opérations sur la machine) et la configuration. Si ce n'est pas le cas, le chemin vers le fichier qui pose problème sera montré. Veuillez vous assurer que Python et/ou Apache peuvent écrire sur ce fichier. Par par exemple sur Ubuntu où Apache correspond à un compte utilisateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre> Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connexion via la librairie ODBC) et que vous ne souhaitez pas utiliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] 86ec2f5f59d5c759cf5980ecc9c26110acce1114 536 535 2012-04-10T15:57:45Z FranckTheeten 15 /* Apache */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes en fonction de votre degré de familiarité avec Python et Apache. Si vous êtes un débutant, veuillez lire le Guide du Débutant [[BeginnersGuide|Beginner’s Guide]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit soit être à l'extérieur d'un pare-feu institutionnel ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de satisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine à la base de données à publier - soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL Server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fonctionner également avec les versions 2.6 et 2.7, la version recommandée est de 2.5 Python (en fait la dernière sous-version stable de la branche 2.5: 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le bib site Web (par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous faites l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier Program Files par défaut. En effet à cet endroit le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir du Subversion du BGBM (Subversion ou SVN est un logiciel qui permet à plusieurs programmeurs de travailler en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la documentation des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (pré-installé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS P Oar exemple si votre provider est installé à l'emplacement ''D:\\biocase'' tapez <pre>D:</pre><pre>cd D://biocase</pre> Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Ci dessous un exemple sous DOs pour Windows, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python figurant dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de interpréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecture depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur en permettant au monde extérieur de voir vos fichiers de configuration , qui incluent des mots de passe. Par exemple, si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre serveur est ''d:\biocase\www'' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache (mais sous Ubuntu;, il est placé dans ''/etc/apache2/sites-available'' et lié sous forme de raccourci à ''/etc/apache2/sites-enabled''). Si vous installez BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez dans tous les cas en informer l'administrateur avant de modifier ce fichier! [[image:instHttpdConf.png|480px]] Sauvez le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utilisez IIS, vous devez créer d'abord un répertoire virtuel pour le chemin physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'). Utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier '''.cgi''' et '''.py''' avec le chemin de votre fichier exécutable Python. Vous devriez ajouter également ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmoins de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'accueil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du type de logiciel de la base de données à publier ainsi que des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'accueil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes). Cette page indique également si cette librairie est présente sur le système et, si dans ce cas, avec quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci: [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe proprement dites sont déjà inclues dans le fichier d'archive que vous avez installé ou bien dans le répertoire Subversion d'où vous êtes parti. La librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos donnes sur le Web). Si vous n'envisagez pas de l'utiliser, il n'est pas nécessaire de l'installer. Mais sinon cliquez sur le lien 'Download', trouvez le fichier d'installation qui convient à votre système d'exploitation, et installez 4SuiteXML à partir du fichier téléchargé. Pour chaque type de base de données (SGBDR ou Système Gestionnaire be base de données Relationnelle dans le jargon), vous devrez toujours installer le package ou driver Python correspondant. Ce driver permet à un programme Python de se connecter à une base de données. Il doit être installé et configuré non pas au niveau de BioCASe, mais au niveau de l'environnement Python. Le BPS peut toutefois vous guider dans leur installation. Si votre collection est par exemple stockée dans une base de données 'MySQL', cliquez sur le lien pour télécharger le module 'MySQL', trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraires dans votre navigateur. La ligne de la liste qui correspond au driver devrait passer de l'orange pour 'non installé' à une indication verte mentionnant la version de la libraire. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utilisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]], actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessaire d'installer Graphviz. Dans le cas contraire nous vous recommandons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquat, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == A la fin de la procédure de configuration, vous devriez changer le mot de passe. L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les paramètres du BPS, dont le mot de passe. Retournez à l'écran d'accueil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la page 'Configuration' dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation. Nous supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''. Vous pouvez encoder un nouveau mot de passe à la quatrièrme ligne de cet écran. Soyez précautionneux en le tapant. Si jamais vous l'oubliez, il est toutefois possible de le réinitialiser en modifiant directement les fichiers de configuration. Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers binaires, car Graphviz n'est pas une librairie Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sur votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary</small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''). Sur un système Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connexion à une base de données), vous devriez vous assurer que tout est installé correctement: # Tapez ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à votre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modifiables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les 'Logs' (fichiers qui enregistrent les opérations sur la machine) et la configuration. Si ce n'est pas le cas, le chemin vers le fichier qui pose problème sera montré. Veuillez vous assurer que Python et/ou Apache peuvent écrire sur ce fichier. Par par exemple sur Ubuntu où Apache correspond à un compte utilisateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre> Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connexion via la librairie ODBC) et que vous ne souhaitez pas utiliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] aa93afff5c97d0b32cac79ea3e752a95a4abc2d6 537 536 2012-04-10T15:58:44Z FranckTheeten 15 /* Apache */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes en fonction de votre degré de familiarité avec Python et Apache. Si vous êtes un débutant, veuillez lire le Guide du Débutant [[BeginnersGuide|Beginner’s Guide]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit soit être à l'extérieur d'un pare-feu institutionnel ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de satisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine à la base de données à publier - soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL Server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fonctionner également avec les versions 2.6 et 2.7, la version recommandée est de 2.5 Python (en fait la dernière sous-version stable de la branche 2.5: 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le bib site Web (par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous faites l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier Program Files par défaut. En effet à cet endroit le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir du Subversion du BGBM (Subversion ou SVN est un logiciel qui permet à plusieurs programmeurs de travailler en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la documentation des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (pré-installé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS P Oar exemple si votre provider est installé à l'emplacement ''D:\\biocase'' tapez <pre>D:</pre><pre>cd D://biocase</pre> Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Ci dessous un exemple sous DOs pour Windows, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python figurant dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de interpréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecture depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur en permettant au monde extérieur de voir vos fichiers de configuration , qui incluent des mots de passe. Par exemple, si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre serveur est ''d:\biocase\www'' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache (mais sous Ubuntu;, il est placé dans ''/etc/apache2/sites-available'' et lié sous forme de raccourci à ''/etc/apache2/sites-enabled''). Si vous installez BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez dans tous les cas en informer l'administrateur avant de modifier ce fichier, car il s'agît du fichier principal qui règle le comportement du serveur! [[image:instHttpdConf.png|480px]] Sauvez le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utilisez IIS, vous devez créer d'abord un répertoire virtuel pour le chemin physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'). Utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier '''.cgi''' et '''.py''' avec le chemin de votre fichier exécutable Python. Vous devriez ajouter également ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmoins de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'accueil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du type de logiciel de la base de données à publier ainsi que des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'accueil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes). Cette page indique également si cette librairie est présente sur le système et, si dans ce cas, avec quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci: [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe proprement dites sont déjà inclues dans le fichier d'archive que vous avez installé ou bien dans le répertoire Subversion d'où vous êtes parti. La librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos donnes sur le Web). Si vous n'envisagez pas de l'utiliser, il n'est pas nécessaire de l'installer. Mais sinon cliquez sur le lien 'Download', trouvez le fichier d'installation qui convient à votre système d'exploitation, et installez 4SuiteXML à partir du fichier téléchargé. Pour chaque type de base de données (SGBDR ou Système Gestionnaire be base de données Relationnelle dans le jargon), vous devrez toujours installer le package ou driver Python correspondant. Ce driver permet à un programme Python de se connecter à une base de données. Il doit être installé et configuré non pas au niveau de BioCASe, mais au niveau de l'environnement Python. Le BPS peut toutefois vous guider dans leur installation. Si votre collection est par exemple stockée dans une base de données 'MySQL', cliquez sur le lien pour télécharger le module 'MySQL', trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraires dans votre navigateur. La ligne de la liste qui correspond au driver devrait passer de l'orange pour 'non installé' à une indication verte mentionnant la version de la libraire. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utilisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]], actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessaire d'installer Graphviz. Dans le cas contraire nous vous recommandons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquat, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == A la fin de la procédure de configuration, vous devriez changer le mot de passe. L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les paramètres du BPS, dont le mot de passe. Retournez à l'écran d'accueil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la page 'Configuration' dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation. Nous supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''. Vous pouvez encoder un nouveau mot de passe à la quatrièrme ligne de cet écran. Soyez précautionneux en le tapant. Si jamais vous l'oubliez, il est toutefois possible de le réinitialiser en modifiant directement les fichiers de configuration. Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers binaires, car Graphviz n'est pas une librairie Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sur votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary</small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''). Sur un système Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connexion à une base de données), vous devriez vous assurer que tout est installé correctement: # Tapez ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à votre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modifiables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les 'Logs' (fichiers qui enregistrent les opérations sur la machine) et la configuration. Si ce n'est pas le cas, le chemin vers le fichier qui pose problème sera montré. Veuillez vous assurer que Python et/ou Apache peuvent écrire sur ce fichier. Par par exemple sur Ubuntu où Apache correspond à un compte utilisateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre> Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connexion via la librairie ODBC) et que vous ne souhaitez pas utiliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] 72854aea4a8d0171a006511a18710e1125493c23 538 537 2012-04-10T16:03:36Z FranckTheeten 15 wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes en fonction de votre degré de familiarité avec Python et Apache. Si vous êtes un débutant, veuillez lire le [[BeginnersGuide_(French)|Guide du Débutant]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit soit être à l'extérieur d'un pare-feu institutionnel ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de satisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine à la base de données à publier - soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL Server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fonctionner également avec les versions 2.6 et 2.7, la version recommandée est de 2.5 Python (en fait la dernière sous-version stable de la branche 2.5: 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le bib site Web (par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous faites l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier Program Files par défaut. En effet à cet endroit le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir du Subversion du BGBM (Subversion ou SVN est un logiciel qui permet à plusieurs programmeurs de travailler en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la documentation des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (pré-installé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS P Oar exemple si votre provider est installé à l'emplacement ''D:\\biocase'' tapez <pre>D:</pre><pre>cd D://biocase</pre> Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Ci dessous un exemple sous DOs pour Windows, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python figurant dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de interpréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecture depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur en permettant au monde extérieur de voir vos fichiers de configuration , qui incluent des mots de passe. Par exemple, si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre serveur est ''d:\biocase\www'' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache (mais sous Ubuntu;, il est placé dans ''/etc/apache2/sites-available'' et lié sous forme de raccourci à ''/etc/apache2/sites-enabled''). Si vous installez BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez dans tous les cas en informer l'administrateur avant de modifier ce fichier, car il s'agît du fichier principal qui règle le comportement du serveur! [[image:instHttpdConf.png|480px]] Sauvez le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utilisez IIS, vous devez créer d'abord un répertoire virtuel pour le chemin physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'). Utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier '''.cgi''' et '''.py''' avec le chemin de votre fichier exécutable Python. Vous devriez ajouter également ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmoins de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'accueil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du type de logiciel de la base de données à publier ainsi que des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'accueil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes). Cette page indique également si cette librairie est présente sur le système et, si dans ce cas, avec quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci: [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe proprement dites sont déjà inclues dans le fichier d'archive que vous avez installé ou bien dans le répertoire Subversion d'où vous êtes parti. La librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos donnes sur le Web). Si vous n'envisagez pas de l'utiliser, il n'est pas nécessaire de l'installer. Mais sinon cliquez sur le lien 'Download', trouvez le fichier d'installation qui convient à votre système d'exploitation, et installez 4SuiteXML à partir du fichier téléchargé. Pour chaque type de base de données (SGBDR ou Système Gestionnaire be base de données Relationnelle dans le jargon), vous devrez toujours installer le package ou driver Python correspondant. Ce driver permet à un programme Python de se connecter à une base de données. Il doit être installé et configuré non pas au niveau de BioCASe, mais au niveau de l'environnement Python. Le BPS peut toutefois vous guider dans leur installation. Si votre collection est par exemple stockée dans une base de données 'MySQL', cliquez sur le lien pour télécharger le module 'MySQL', trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraires dans votre navigateur. La ligne de la liste qui correspond au driver devrait passer de l'orange pour 'non installé' à une indication verte mentionnant la version de la libraire. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utilisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]], actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessaire d'installer Graphviz. Dans le cas contraire nous vous recommandons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquat, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == A la fin de la procédure de configuration, vous devriez changer le mot de passe. L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les paramètres du BPS, dont le mot de passe. Retournez à l'écran d'accueil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la page 'Configuration' dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation. Nous supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''. Vous pouvez encoder un nouveau mot de passe à la quatrièrme ligne de cet écran. Soyez précautionneux en le tapant. Si jamais vous l'oubliez, il est toutefois possible de le réinitialiser en modifiant directement les fichiers de configuration. Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers binaires, car Graphviz n'est pas une librairie Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sur votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary</small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''). Sur un système Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connexion à une base de données), vous devriez vous assurer que tout est installé correctement: # Tapez ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à votre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modifiables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les 'Logs' (fichiers qui enregistrent les opérations sur la machine) et la configuration. Si ce n'est pas le cas, le chemin vers le fichier qui pose problème sera montré. Veuillez vous assurer que Python et/ou Apache peuvent écrire sur ce fichier. Par par exemple sur Ubuntu où Apache correspond à un compte utilisateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre> Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connexion via la librairie ODBC) et que vous ne souhaitez pas utiliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] 028c77d345e62eae1b7a38da6a392cb2a613fd07 539 538 2012-04-10T16:04:15Z FranckTheeten 15 /* Prérequis */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes en fonction de votre degré de familiarité avec Python et Apache. Si vous êtes un débutant, veuillez lire le [[BeginnersGuide_(French)|Guide du Débutant]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit, soit être à l'extérieur d'un pare-feu institutionnel, ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de satisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine à la base de données à publier - soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL Server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fonctionner également avec les versions 2.6 et 2.7, la version recommandée est de 2.5 Python (en fait la dernière sous-version stable de la branche 2.5: 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le bib site Web (par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous faites l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier Program Files par défaut. En effet à cet endroit le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir du Subversion du BGBM (Subversion ou SVN est un logiciel qui permet à plusieurs programmeurs de travailler en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la documentation des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (pré-installé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS P Oar exemple si votre provider est installé à l'emplacement ''D:\\biocase'' tapez <pre>D:</pre><pre>cd D://biocase</pre> Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Ci dessous un exemple sous DOs pour Windows, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python figurant dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de interpréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecture depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur en permettant au monde extérieur de voir vos fichiers de configuration , qui incluent des mots de passe. Par exemple, si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre serveur est ''d:\biocase\www'' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache (mais sous Ubuntu;, il est placé dans ''/etc/apache2/sites-available'' et lié sous forme de raccourci à ''/etc/apache2/sites-enabled''). Si vous installez BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez dans tous les cas en informer l'administrateur avant de modifier ce fichier, car il s'agît du fichier principal qui règle le comportement du serveur! [[image:instHttpdConf.png|480px]] Sauvez le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utilisez IIS, vous devez créer d'abord un répertoire virtuel pour le chemin physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'). Utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier '''.cgi''' et '''.py''' avec le chemin de votre fichier exécutable Python. Vous devriez ajouter également ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmoins de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'accueil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du type de logiciel de la base de données à publier ainsi que des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'accueil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes). Cette page indique également si cette librairie est présente sur le système et, si dans ce cas, avec quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci: [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe proprement dites sont déjà inclues dans le fichier d'archive que vous avez installé ou bien dans le répertoire Subversion d'où vous êtes parti. La librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos donnes sur le Web). Si vous n'envisagez pas de l'utiliser, il n'est pas nécessaire de l'installer. Mais sinon cliquez sur le lien 'Download', trouvez le fichier d'installation qui convient à votre système d'exploitation, et installez 4SuiteXML à partir du fichier téléchargé. Pour chaque type de base de données (SGBDR ou Système Gestionnaire be base de données Relationnelle dans le jargon), vous devrez toujours installer le package ou driver Python correspondant. Ce driver permet à un programme Python de se connecter à une base de données. Il doit être installé et configuré non pas au niveau de BioCASe, mais au niveau de l'environnement Python. Le BPS peut toutefois vous guider dans leur installation. Si votre collection est par exemple stockée dans une base de données 'MySQL', cliquez sur le lien pour télécharger le module 'MySQL', trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraires dans votre navigateur. La ligne de la liste qui correspond au driver devrait passer de l'orange pour 'non installé' à une indication verte mentionnant la version de la libraire. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utilisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]], actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessaire d'installer Graphviz. Dans le cas contraire nous vous recommandons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquat, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == A la fin de la procédure de configuration, vous devriez changer le mot de passe. L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les paramètres du BPS, dont le mot de passe. Retournez à l'écran d'accueil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la page 'Configuration' dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation. Nous supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''. Vous pouvez encoder un nouveau mot de passe à la quatrièrme ligne de cet écran. Soyez précautionneux en le tapant. Si jamais vous l'oubliez, il est toutefois possible de le réinitialiser en modifiant directement les fichiers de configuration. Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers binaires, car Graphviz n'est pas une librairie Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sur votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary</small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''). Sur un système Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connexion à une base de données), vous devriez vous assurer que tout est installé correctement: # Tapez ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à votre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modifiables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les 'Logs' (fichiers qui enregistrent les opérations sur la machine) et la configuration. Si ce n'est pas le cas, le chemin vers le fichier qui pose problème sera montré. Veuillez vous assurer que Python et/ou Apache peuvent écrire sur ce fichier. Par par exemple sur Ubuntu où Apache correspond à un compte utilisateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre> Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connexion via la librairie ODBC) et que vous ne souhaitez pas utiliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] 5a49a334a48ed30478991ca3b620a2c38e068ef5 540 539 2012-04-10T16:05:00Z FranckTheeten 15 /* Prérequis */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes en fonction de votre degré de familiarité avec Python et Apache. Si vous êtes un débutant, veuillez lire le [[BeginnersGuide_(French)|Guide du Débutant]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit, soit être à l'extérieur d'un pare-feu institutionnel, ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de satisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine vers la base de données à publier, soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL Server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fonctionner également avec les versions 2.6 et 2.7, la version recommandée est de 2.5 Python (en fait la dernière sous-version stable de la branche 2.5: 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le bib site Web (par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous faites l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier Program Files par défaut. En effet à cet endroit le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir du Subversion du BGBM (Subversion ou SVN est un logiciel qui permet à plusieurs programmeurs de travailler en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la documentation des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (pré-installé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS P Oar exemple si votre provider est installé à l'emplacement ''D:\\biocase'' tapez <pre>D:</pre><pre>cd D://biocase</pre> Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Ci dessous un exemple sous DOs pour Windows, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python figurant dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de interpréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecture depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur en permettant au monde extérieur de voir vos fichiers de configuration , qui incluent des mots de passe. Par exemple, si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre serveur est ''d:\biocase\www'' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache (mais sous Ubuntu;, il est placé dans ''/etc/apache2/sites-available'' et lié sous forme de raccourci à ''/etc/apache2/sites-enabled''). Si vous installez BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez dans tous les cas en informer l'administrateur avant de modifier ce fichier, car il s'agît du fichier principal qui règle le comportement du serveur! [[image:instHttpdConf.png|480px]] Sauvez le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utilisez IIS, vous devez créer d'abord un répertoire virtuel pour le chemin physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'). Utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier '''.cgi''' et '''.py''' avec le chemin de votre fichier exécutable Python. Vous devriez ajouter également ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmoins de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'accueil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du type de logiciel de la base de données à publier ainsi que des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'accueil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes). Cette page indique également si cette librairie est présente sur le système et, si dans ce cas, avec quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci: [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe proprement dites sont déjà inclues dans le fichier d'archive que vous avez installé ou bien dans le répertoire Subversion d'où vous êtes parti. La librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos donnes sur le Web). Si vous n'envisagez pas de l'utiliser, il n'est pas nécessaire de l'installer. Mais sinon cliquez sur le lien 'Download', trouvez le fichier d'installation qui convient à votre système d'exploitation, et installez 4SuiteXML à partir du fichier téléchargé. Pour chaque type de base de données (SGBDR ou Système Gestionnaire be base de données Relationnelle dans le jargon), vous devrez toujours installer le package ou driver Python correspondant. Ce driver permet à un programme Python de se connecter à une base de données. Il doit être installé et configuré non pas au niveau de BioCASe, mais au niveau de l'environnement Python. Le BPS peut toutefois vous guider dans leur installation. Si votre collection est par exemple stockée dans une base de données 'MySQL', cliquez sur le lien pour télécharger le module 'MySQL', trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraires dans votre navigateur. La ligne de la liste qui correspond au driver devrait passer de l'orange pour 'non installé' à une indication verte mentionnant la version de la libraire. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utilisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]], actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessaire d'installer Graphviz. Dans le cas contraire nous vous recommandons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquat, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == A la fin de la procédure de configuration, vous devriez changer le mot de passe. L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les paramètres du BPS, dont le mot de passe. Retournez à l'écran d'accueil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la page 'Configuration' dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation. Nous supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''. Vous pouvez encoder un nouveau mot de passe à la quatrièrme ligne de cet écran. Soyez précautionneux en le tapant. Si jamais vous l'oubliez, il est toutefois possible de le réinitialiser en modifiant directement les fichiers de configuration. Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers binaires, car Graphviz n'est pas une librairie Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sur votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary</small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''). Sur un système Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connexion à une base de données), vous devriez vous assurer que tout est installé correctement: # Tapez ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à votre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modifiables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les 'Logs' (fichiers qui enregistrent les opérations sur la machine) et la configuration. Si ce n'est pas le cas, le chemin vers le fichier qui pose problème sera montré. Veuillez vous assurer que Python et/ou Apache peuvent écrire sur ce fichier. Par par exemple sur Ubuntu où Apache correspond à un compte utilisateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre> Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connexion via la librairie ODBC) et que vous ne souhaitez pas utiliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] f37e8511def710e7836359d8f85e92726673634c 541 540 2012-04-10T16:06:12Z FranckTheeten 15 /* Installation de Apache / IIS et Python */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes en fonction de votre degré de familiarité avec Python et Apache. Si vous êtes un débutant, veuillez lire le [[BeginnersGuide_(French)|Guide du Débutant]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit, soit être à l'extérieur d'un pare-feu institutionnel, ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de satisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine vers la base de données à publier, soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL Server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système que vous utilisez. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fonctionner également avec les versions 2.6 et 2.7, la version recommandée est de 2.5 Python (en fait la dernière sous-version stable de la branche 2.5: 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le bib site Web (par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous faites l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier Program Files par défaut. En effet à cet endroit le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir du Subversion du BGBM (Subversion ou SVN est un logiciel qui permet à plusieurs programmeurs de travailler en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la documentation des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (pré-installé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS P Oar exemple si votre provider est installé à l'emplacement ''D:\\biocase'' tapez <pre>D:</pre><pre>cd D://biocase</pre> Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Ci dessous un exemple sous DOs pour Windows, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python figurant dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de interpréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecture depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur en permettant au monde extérieur de voir vos fichiers de configuration , qui incluent des mots de passe. Par exemple, si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre serveur est ''d:\biocase\www'' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache (mais sous Ubuntu;, il est placé dans ''/etc/apache2/sites-available'' et lié sous forme de raccourci à ''/etc/apache2/sites-enabled''). Si vous installez BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez dans tous les cas en informer l'administrateur avant de modifier ce fichier, car il s'agît du fichier principal qui règle le comportement du serveur! [[image:instHttpdConf.png|480px]] Sauvez le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utilisez IIS, vous devez créer d'abord un répertoire virtuel pour le chemin physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'). Utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier '''.cgi''' et '''.py''' avec le chemin de votre fichier exécutable Python. Vous devriez ajouter également ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmoins de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'accueil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du type de logiciel de la base de données à publier ainsi que des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'accueil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes). Cette page indique également si cette librairie est présente sur le système et, si dans ce cas, avec quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci: [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe proprement dites sont déjà inclues dans le fichier d'archive que vous avez installé ou bien dans le répertoire Subversion d'où vous êtes parti. La librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos donnes sur le Web). Si vous n'envisagez pas de l'utiliser, il n'est pas nécessaire de l'installer. Mais sinon cliquez sur le lien 'Download', trouvez le fichier d'installation qui convient à votre système d'exploitation, et installez 4SuiteXML à partir du fichier téléchargé. Pour chaque type de base de données (SGBDR ou Système Gestionnaire be base de données Relationnelle dans le jargon), vous devrez toujours installer le package ou driver Python correspondant. Ce driver permet à un programme Python de se connecter à une base de données. Il doit être installé et configuré non pas au niveau de BioCASe, mais au niveau de l'environnement Python. Le BPS peut toutefois vous guider dans leur installation. Si votre collection est par exemple stockée dans une base de données 'MySQL', cliquez sur le lien pour télécharger le module 'MySQL', trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraires dans votre navigateur. La ligne de la liste qui correspond au driver devrait passer de l'orange pour 'non installé' à une indication verte mentionnant la version de la libraire. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utilisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]], actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessaire d'installer Graphviz. Dans le cas contraire nous vous recommandons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquat, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == A la fin de la procédure de configuration, vous devriez changer le mot de passe. L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les paramètres du BPS, dont le mot de passe. Retournez à l'écran d'accueil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la page 'Configuration' dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation. Nous supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''. Vous pouvez encoder un nouveau mot de passe à la quatrièrme ligne de cet écran. Soyez précautionneux en le tapant. Si jamais vous l'oubliez, il est toutefois possible de le réinitialiser en modifiant directement les fichiers de configuration. Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers binaires, car Graphviz n'est pas une librairie Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sur votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary</small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''). Sur un système Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connexion à une base de données), vous devriez vous assurer que tout est installé correctement: # Tapez ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à votre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modifiables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les 'Logs' (fichiers qui enregistrent les opérations sur la machine) et la configuration. Si ce n'est pas le cas, le chemin vers le fichier qui pose problème sera montré. Veuillez vous assurer que Python et/ou Apache peuvent écrire sur ce fichier. Par par exemple sur Ubuntu où Apache correspond à un compte utilisateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre> Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connexion via la librairie ODBC) et que vous ne souhaitez pas utiliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] c2e3187a1cc31d3ac1e5ddfa098e086cbb25b4a8 542 541 2012-04-10T16:07:01Z FranckTheeten 15 /* Installation de Apache / IIS et Python */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes en fonction de votre degré de familiarité avec Python et Apache. Si vous êtes un débutant, veuillez lire le [[BeginnersGuide_(French)|Guide du Débutant]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit, soit être à l'extérieur d'un pare-feu institutionnel, ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de satisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine vers la base de données à publier, soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL Server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système que vous utilisez. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fonctionner également avec les versions 2.6 et 2.7, la version recommandée est Python 2.5 (en fait il s'agît plutôt de la dernière sous-version stable de la branche 2.5, que nous notons 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le bib site Web (par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous faites l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier Program Files par défaut. En effet à cet endroit le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir du Subversion du BGBM (Subversion ou SVN est un logiciel qui permet à plusieurs programmeurs de travailler en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la documentation des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (pré-installé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS P Oar exemple si votre provider est installé à l'emplacement ''D:\\biocase'' tapez <pre>D:</pre><pre>cd D://biocase</pre> Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Ci dessous un exemple sous DOs pour Windows, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python figurant dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de interpréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecture depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur en permettant au monde extérieur de voir vos fichiers de configuration , qui incluent des mots de passe. Par exemple, si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre serveur est ''d:\biocase\www'' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache (mais sous Ubuntu;, il est placé dans ''/etc/apache2/sites-available'' et lié sous forme de raccourci à ''/etc/apache2/sites-enabled''). Si vous installez BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez dans tous les cas en informer l'administrateur avant de modifier ce fichier, car il s'agît du fichier principal qui règle le comportement du serveur! [[image:instHttpdConf.png|480px]] Sauvez le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utilisez IIS, vous devez créer d'abord un répertoire virtuel pour le chemin physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'). Utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier '''.cgi''' et '''.py''' avec le chemin de votre fichier exécutable Python. Vous devriez ajouter également ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmoins de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'accueil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du type de logiciel de la base de données à publier ainsi que des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'accueil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes). Cette page indique également si cette librairie est présente sur le système et, si dans ce cas, avec quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci: [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe proprement dites sont déjà inclues dans le fichier d'archive que vous avez installé ou bien dans le répertoire Subversion d'où vous êtes parti. La librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos donnes sur le Web). Si vous n'envisagez pas de l'utiliser, il n'est pas nécessaire de l'installer. Mais sinon cliquez sur le lien 'Download', trouvez le fichier d'installation qui convient à votre système d'exploitation, et installez 4SuiteXML à partir du fichier téléchargé. Pour chaque type de base de données (SGBDR ou Système Gestionnaire be base de données Relationnelle dans le jargon), vous devrez toujours installer le package ou driver Python correspondant. Ce driver permet à un programme Python de se connecter à une base de données. Il doit être installé et configuré non pas au niveau de BioCASe, mais au niveau de l'environnement Python. Le BPS peut toutefois vous guider dans leur installation. Si votre collection est par exemple stockée dans une base de données 'MySQL', cliquez sur le lien pour télécharger le module 'MySQL', trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraires dans votre navigateur. La ligne de la liste qui correspond au driver devrait passer de l'orange pour 'non installé' à une indication verte mentionnant la version de la libraire. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utilisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]], actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessaire d'installer Graphviz. Dans le cas contraire nous vous recommandons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquat, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == A la fin de la procédure de configuration, vous devriez changer le mot de passe. L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les paramètres du BPS, dont le mot de passe. Retournez à l'écran d'accueil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la page 'Configuration' dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation. Nous supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''. Vous pouvez encoder un nouveau mot de passe à la quatrièrme ligne de cet écran. Soyez précautionneux en le tapant. Si jamais vous l'oubliez, il est toutefois possible de le réinitialiser en modifiant directement les fichiers de configuration. Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers binaires, car Graphviz n'est pas une librairie Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sur votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary</small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''). Sur un système Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connexion à une base de données), vous devriez vous assurer que tout est installé correctement: # Tapez ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à votre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modifiables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les 'Logs' (fichiers qui enregistrent les opérations sur la machine) et la configuration. Si ce n'est pas le cas, le chemin vers le fichier qui pose problème sera montré. Veuillez vous assurer que Python et/ou Apache peuvent écrire sur ce fichier. Par par exemple sur Ubuntu où Apache correspond à un compte utilisateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre> Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connexion via la librairie ODBC) et que vous ne souhaitez pas utiliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] a7e4ff014ddc3eff4bb41390696e186ec852b56e 544 542 2012-04-10T16:09:28Z FranckTheeten 15 /* Installation de Apache / IIS et Python */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes en fonction de votre degré de familiarité avec Python et Apache. Si vous êtes un débutant, veuillez lire le [[BeginnersGuide_(French)|Guide du Débutant]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit, soit être à l'extérieur d'un pare-feu institutionnel, ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de satisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine vers la base de données à publier, soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL Server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système que vous utilisez. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fonctionner également avec les versions 2.6 et 2.7, la version recommandée est Python 2.5 (en fait il s'agît plutôt de la dernière sous-version stable de la branche 2.5, que nous notons 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le site Web du driver(par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous effectuez l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier ''C:/Program Files'' par défaut. En effet, lorsqu'il est placé à cet endroit, le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit comme une archive zippée g-ou directement à partir du Subversion du BGBM (Subversion ou SVN est un logiciel qui permet à plusieurs programmeurs de travailler en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la documentation des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (pré-installé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS P Oar exemple si votre provider est installé à l'emplacement ''D:\\biocase'' tapez <pre>D:</pre><pre>cd D://biocase</pre> Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Ci dessous un exemple sous DOs pour Windows, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python figurant dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de interpréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecture depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur en permettant au monde extérieur de voir vos fichiers de configuration , qui incluent des mots de passe. Par exemple, si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre serveur est ''d:\biocase\www'' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache (mais sous Ubuntu;, il est placé dans ''/etc/apache2/sites-available'' et lié sous forme de raccourci à ''/etc/apache2/sites-enabled''). Si vous installez BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez dans tous les cas en informer l'administrateur avant de modifier ce fichier, car il s'agît du fichier principal qui règle le comportement du serveur! [[image:instHttpdConf.png|480px]] Sauvez le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utilisez IIS, vous devez créer d'abord un répertoire virtuel pour le chemin physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'). Utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier '''.cgi''' et '''.py''' avec le chemin de votre fichier exécutable Python. Vous devriez ajouter également ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmoins de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'accueil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du type de logiciel de la base de données à publier ainsi que des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'accueil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes). Cette page indique également si cette librairie est présente sur le système et, si dans ce cas, avec quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci: [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe proprement dites sont déjà inclues dans le fichier d'archive que vous avez installé ou bien dans le répertoire Subversion d'où vous êtes parti. La librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos donnes sur le Web). Si vous n'envisagez pas de l'utiliser, il n'est pas nécessaire de l'installer. Mais sinon cliquez sur le lien 'Download', trouvez le fichier d'installation qui convient à votre système d'exploitation, et installez 4SuiteXML à partir du fichier téléchargé. Pour chaque type de base de données (SGBDR ou Système Gestionnaire be base de données Relationnelle dans le jargon), vous devrez toujours installer le package ou driver Python correspondant. Ce driver permet à un programme Python de se connecter à une base de données. Il doit être installé et configuré non pas au niveau de BioCASe, mais au niveau de l'environnement Python. Le BPS peut toutefois vous guider dans leur installation. Si votre collection est par exemple stockée dans une base de données 'MySQL', cliquez sur le lien pour télécharger le module 'MySQL', trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraires dans votre navigateur. La ligne de la liste qui correspond au driver devrait passer de l'orange pour 'non installé' à une indication verte mentionnant la version de la libraire. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utilisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]], actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessaire d'installer Graphviz. Dans le cas contraire nous vous recommandons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquat, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == A la fin de la procédure de configuration, vous devriez changer le mot de passe. L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les paramètres du BPS, dont le mot de passe. Retournez à l'écran d'accueil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la page 'Configuration' dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation. Nous supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''. Vous pouvez encoder un nouveau mot de passe à la quatrièrme ligne de cet écran. Soyez précautionneux en le tapant. Si jamais vous l'oubliez, il est toutefois possible de le réinitialiser en modifiant directement les fichiers de configuration. Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers binaires, car Graphviz n'est pas une librairie Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sur votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary</small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''). Sur un système Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connexion à une base de données), vous devriez vous assurer que tout est installé correctement: # Tapez ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à votre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modifiables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les 'Logs' (fichiers qui enregistrent les opérations sur la machine) et la configuration. Si ce n'est pas le cas, le chemin vers le fichier qui pose problème sera montré. Veuillez vous assurer que Python et/ou Apache peuvent écrire sur ce fichier. Par par exemple sur Ubuntu où Apache correspond à un compte utilisateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre> Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connexion via la librairie ODBC) et que vous ne souhaitez pas utiliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] 86f0b0cd3c297a8def68700f2e5aa082d63af311 545 544 2012-04-10T16:10:04Z FranckTheeten 15 /* Téléchargement du logiciel fournisseur */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes en fonction de votre degré de familiarité avec Python et Apache. Si vous êtes un débutant, veuillez lire le [[BeginnersGuide_(French)|Guide du Débutant]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit, soit être à l'extérieur d'un pare-feu institutionnel, ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de satisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine vers la base de données à publier, soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL Server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système que vous utilisez. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fonctionner également avec les versions 2.6 et 2.7, la version recommandée est Python 2.5 (en fait il s'agît plutôt de la dernière sous-version stable de la branche 2.5, que nous notons 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le site Web du driver(par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous effectuez l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier ''C:/Program Files'' par défaut. En effet, lorsqu'il est placé à cet endroit, le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit sous forme d'archive zippée, ou directement à partir du Subversion du BGBM (Subversion ou ''SVN'' est un logiciel qui permet à plusieurs programmeurs de travailler en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la documentation des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion avant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production, qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (pré-installé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS P Oar exemple si votre provider est installé à l'emplacement ''D:\\biocase'' tapez <pre>D:</pre><pre>cd D://biocase</pre> Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Ci dessous un exemple sous DOs pour Windows, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python figurant dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de interpréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecture depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur en permettant au monde extérieur de voir vos fichiers de configuration , qui incluent des mots de passe. Par exemple, si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre serveur est ''d:\biocase\www'' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache (mais sous Ubuntu;, il est placé dans ''/etc/apache2/sites-available'' et lié sous forme de raccourci à ''/etc/apache2/sites-enabled''). Si vous installez BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez dans tous les cas en informer l'administrateur avant de modifier ce fichier, car il s'agît du fichier principal qui règle le comportement du serveur! [[image:instHttpdConf.png|480px]] Sauvez le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utilisez IIS, vous devez créer d'abord un répertoire virtuel pour le chemin physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'). Utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier '''.cgi''' et '''.py''' avec le chemin de votre fichier exécutable Python. Vous devriez ajouter également ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmoins de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'accueil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du type de logiciel de la base de données à publier ainsi que des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'accueil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes). Cette page indique également si cette librairie est présente sur le système et, si dans ce cas, avec quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci: [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe proprement dites sont déjà inclues dans le fichier d'archive que vous avez installé ou bien dans le répertoire Subversion d'où vous êtes parti. La librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos donnes sur le Web). Si vous n'envisagez pas de l'utiliser, il n'est pas nécessaire de l'installer. Mais sinon cliquez sur le lien 'Download', trouvez le fichier d'installation qui convient à votre système d'exploitation, et installez 4SuiteXML à partir du fichier téléchargé. Pour chaque type de base de données (SGBDR ou Système Gestionnaire be base de données Relationnelle dans le jargon), vous devrez toujours installer le package ou driver Python correspondant. Ce driver permet à un programme Python de se connecter à une base de données. Il doit être installé et configuré non pas au niveau de BioCASe, mais au niveau de l'environnement Python. Le BPS peut toutefois vous guider dans leur installation. Si votre collection est par exemple stockée dans une base de données 'MySQL', cliquez sur le lien pour télécharger le module 'MySQL', trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraires dans votre navigateur. La ligne de la liste qui correspond au driver devrait passer de l'orange pour 'non installé' à une indication verte mentionnant la version de la libraire. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utilisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]], actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessaire d'installer Graphviz. Dans le cas contraire nous vous recommandons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquat, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == A la fin de la procédure de configuration, vous devriez changer le mot de passe. L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les paramètres du BPS, dont le mot de passe. Retournez à l'écran d'accueil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la page 'Configuration' dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation. Nous supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''. Vous pouvez encoder un nouveau mot de passe à la quatrièrme ligne de cet écran. Soyez précautionneux en le tapant. Si jamais vous l'oubliez, il est toutefois possible de le réinitialiser en modifiant directement les fichiers de configuration. Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers binaires, car Graphviz n'est pas une librairie Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sur votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary</small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''). Sur un système Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connexion à une base de données), vous devriez vous assurer que tout est installé correctement: # Tapez ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à votre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modifiables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les 'Logs' (fichiers qui enregistrent les opérations sur la machine) et la configuration. Si ce n'est pas le cas, le chemin vers le fichier qui pose problème sera montré. Veuillez vous assurer que Python et/ou Apache peuvent écrire sur ce fichier. Par par exemple sur Ubuntu où Apache correspond à un compte utilisateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre> Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connexion via la librairie ODBC) et que vous ne souhaitez pas utiliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] 89ed90867fbdfd460dcec852fc823b04ed6c309f 546 545 2012-04-10T16:10:47Z FranckTheeten 15 /* Téléchargement du logiciel fournisseur */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes en fonction de votre degré de familiarité avec Python et Apache. Si vous êtes un débutant, veuillez lire le [[BeginnersGuide_(French)|Guide du Débutant]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit, soit être à l'extérieur d'un pare-feu institutionnel, ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de satisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine vers la base de données à publier, soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL Server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système que vous utilisez. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fonctionner également avec les versions 2.6 et 2.7, la version recommandée est Python 2.5 (en fait il s'agît plutôt de la dernière sous-version stable de la branche 2.5, que nous notons 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le site Web du driver(par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous effectuez l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier ''C:/Program Files'' par défaut. En effet, lorsqu'il est placé à cet endroit, le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit sous forme d'archive zippée, ou directement à partir du Subversion du BGBM (Subversion ou ''SVN'' est un logiciel qui permet à plusieurs programmeurs de travailler en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la documentation des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion auparavant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (pré-installé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites ub clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS P Oar exemple si votre provider est installé à l'emplacement ''D:\\biocase'' tapez <pre>D:</pre><pre>cd D://biocase</pre> Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Ci dessous un exemple sous DOs pour Windows, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python figurant dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de interpréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecture depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur en permettant au monde extérieur de voir vos fichiers de configuration , qui incluent des mots de passe. Par exemple, si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre serveur est ''d:\biocase\www'' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache (mais sous Ubuntu;, il est placé dans ''/etc/apache2/sites-available'' et lié sous forme de raccourci à ''/etc/apache2/sites-enabled''). Si vous installez BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez dans tous les cas en informer l'administrateur avant de modifier ce fichier, car il s'agît du fichier principal qui règle le comportement du serveur! [[image:instHttpdConf.png|480px]] Sauvez le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utilisez IIS, vous devez créer d'abord un répertoire virtuel pour le chemin physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'). Utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier '''.cgi''' et '''.py''' avec le chemin de votre fichier exécutable Python. Vous devriez ajouter également ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmoins de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'accueil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du type de logiciel de la base de données à publier ainsi que des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'accueil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes). Cette page indique également si cette librairie est présente sur le système et, si dans ce cas, avec quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci: [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe proprement dites sont déjà inclues dans le fichier d'archive que vous avez installé ou bien dans le répertoire Subversion d'où vous êtes parti. La librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos donnes sur le Web). Si vous n'envisagez pas de l'utiliser, il n'est pas nécessaire de l'installer. Mais sinon cliquez sur le lien 'Download', trouvez le fichier d'installation qui convient à votre système d'exploitation, et installez 4SuiteXML à partir du fichier téléchargé. Pour chaque type de base de données (SGBDR ou Système Gestionnaire be base de données Relationnelle dans le jargon), vous devrez toujours installer le package ou driver Python correspondant. Ce driver permet à un programme Python de se connecter à une base de données. Il doit être installé et configuré non pas au niveau de BioCASe, mais au niveau de l'environnement Python. Le BPS peut toutefois vous guider dans leur installation. Si votre collection est par exemple stockée dans une base de données 'MySQL', cliquez sur le lien pour télécharger le module 'MySQL', trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraires dans votre navigateur. La ligne de la liste qui correspond au driver devrait passer de l'orange pour 'non installé' à une indication verte mentionnant la version de la libraire. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utilisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]], actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessaire d'installer Graphviz. Dans le cas contraire nous vous recommandons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquat, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == A la fin de la procédure de configuration, vous devriez changer le mot de passe. L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les paramètres du BPS, dont le mot de passe. Retournez à l'écran d'accueil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la page 'Configuration' dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation. Nous supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''. Vous pouvez encoder un nouveau mot de passe à la quatrièrme ligne de cet écran. Soyez précautionneux en le tapant. Si jamais vous l'oubliez, il est toutefois possible de le réinitialiser en modifiant directement les fichiers de configuration. Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers binaires, car Graphviz n'est pas une librairie Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sur votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary</small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''). Sur un système Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connexion à une base de données), vous devriez vous assurer que tout est installé correctement: # Tapez ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à votre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modifiables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les 'Logs' (fichiers qui enregistrent les opérations sur la machine) et la configuration. Si ce n'est pas le cas, le chemin vers le fichier qui pose problème sera montré. Veuillez vous assurer que Python et/ou Apache peuvent écrire sur ce fichier. Par par exemple sur Ubuntu où Apache correspond à un compte utilisateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre> Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connexion via la librairie ODBC) et que vous ne souhaitez pas utiliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] 9080c3de4a8a5a8c242070812d29089b01d8d1c1 547 546 2012-04-10T16:12:08Z FranckTheeten 15 /* Téléchargement à partir du dépôt Subversion */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes en fonction de votre degré de familiarité avec Python et Apache. Si vous êtes un débutant, veuillez lire le [[BeginnersGuide_(French)|Guide du Débutant]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit, soit être à l'extérieur d'un pare-feu institutionnel, ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de satisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine vers la base de données à publier, soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL Server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système que vous utilisez. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fonctionner également avec les versions 2.6 et 2.7, la version recommandée est Python 2.5 (en fait il s'agît plutôt de la dernière sous-version stable de la branche 2.5, que nous notons 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le site Web du driver(par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous effectuez l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier ''C:/Program Files'' par défaut. En effet, lorsqu'il est placé à cet endroit, le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur == Vous pouvez télécharger le BPS soit sous forme d'archive zippée, ou directement à partir du Subversion du BGBM (Subversion ou ''SVN'' est un logiciel qui permet à plusieurs programmeurs de travailler en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la documentation des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion auparavant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (pré-installé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites un clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS P Oar exemple si votre provider est installé à l'emplacement ''D:\\biocase'' tapez <pre>D:</pre><pre>cd D://biocase</pre> Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Ci dessous un exemple sous DOs pour Windows, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python figurant dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de interpréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecture depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur en permettant au monde extérieur de voir vos fichiers de configuration , qui incluent des mots de passe. Par exemple, si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre serveur est ''d:\biocase\www'' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache (mais sous Ubuntu;, il est placé dans ''/etc/apache2/sites-available'' et lié sous forme de raccourci à ''/etc/apache2/sites-enabled''). Si vous installez BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez dans tous les cas en informer l'administrateur avant de modifier ce fichier, car il s'agît du fichier principal qui règle le comportement du serveur! [[image:instHttpdConf.png|480px]] Sauvez le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utilisez IIS, vous devez créer d'abord un répertoire virtuel pour le chemin physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'). Utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier '''.cgi''' et '''.py''' avec le chemin de votre fichier exécutable Python. Vous devriez ajouter également ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmoins de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'accueil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du type de logiciel de la base de données à publier ainsi que des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'accueil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes). Cette page indique également si cette librairie est présente sur le système et, si dans ce cas, avec quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci: [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe proprement dites sont déjà inclues dans le fichier d'archive que vous avez installé ou bien dans le répertoire Subversion d'où vous êtes parti. La librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos donnes sur le Web). Si vous n'envisagez pas de l'utiliser, il n'est pas nécessaire de l'installer. Mais sinon cliquez sur le lien 'Download', trouvez le fichier d'installation qui convient à votre système d'exploitation, et installez 4SuiteXML à partir du fichier téléchargé. Pour chaque type de base de données (SGBDR ou Système Gestionnaire be base de données Relationnelle dans le jargon), vous devrez toujours installer le package ou driver Python correspondant. Ce driver permet à un programme Python de se connecter à une base de données. Il doit être installé et configuré non pas au niveau de BioCASe, mais au niveau de l'environnement Python. Le BPS peut toutefois vous guider dans leur installation. Si votre collection est par exemple stockée dans une base de données 'MySQL', cliquez sur le lien pour télécharger le module 'MySQL', trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraires dans votre navigateur. La ligne de la liste qui correspond au driver devrait passer de l'orange pour 'non installé' à une indication verte mentionnant la version de la libraire. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utilisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]], actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessaire d'installer Graphviz. Dans le cas contraire nous vous recommandons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquat, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == A la fin de la procédure de configuration, vous devriez changer le mot de passe. L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les paramètres du BPS, dont le mot de passe. Retournez à l'écran d'accueil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la page 'Configuration' dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation. Nous supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''. Vous pouvez encoder un nouveau mot de passe à la quatrièrme ligne de cet écran. Soyez précautionneux en le tapant. Si jamais vous l'oubliez, il est toutefois possible de le réinitialiser en modifiant directement les fichiers de configuration. Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers binaires, car Graphviz n'est pas une librairie Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sur votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary</small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''). Sur un système Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connexion à une base de données), vous devriez vous assurer que tout est installé correctement: # Tapez ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à votre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modifiables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les 'Logs' (fichiers qui enregistrent les opérations sur la machine) et la configuration. Si ce n'est pas le cas, le chemin vers le fichier qui pose problème sera montré. Veuillez vous assurer que Python et/ou Apache peuvent écrire sur ce fichier. Par par exemple sur Ubuntu où Apache correspond à un compte utilisateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre> Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connexion via la librairie ODBC) et que vous ne souhaitez pas utiliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] c33985cf970d8ec4a3c38078c95e648dadfb30b4 548 547 2012-04-10T16:13:11Z FranckTheeten 15 /* Téléchargement du logiciel fournisseur */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes en fonction de votre degré de familiarité avec Python et Apache. Si vous êtes un débutant, veuillez lire le [[BeginnersGuide_(French)|Guide du Débutant]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit, soit être à l'extérieur d'un pare-feu institutionnel, ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de satisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine vers la base de données à publier, soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL Server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système que vous utilisez. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fonctionner également avec les versions 2.6 et 2.7, la version recommandée est Python 2.5 (en fait il s'agît plutôt de la dernière sous-version stable de la branche 2.5, que nous notons 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le site Web du driver(par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous effectuez l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier ''C:/Program Files'' par défaut. En effet, lorsqu'il est placé à cet endroit, le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur de données BioCASe == Vous pouvez télécharger le BPS soit sous forme d'archive zippée, ou directement à partir du Subversion du BGBM (Subversion ou ''SVN'' est un logiciel qui permet à plusieurs programmeurs de travailler en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la documentation des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion auparavant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (pré-installé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites un clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installayion) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau le script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS P Oar exemple si votre provider est installé à l'emplacement ''D:\\biocase'' tapez <pre>D:</pre><pre>cd D://biocase</pre> Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Ci dessous un exemple sous DOs pour Windows, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python figurant dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de interpréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecture depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur en permettant au monde extérieur de voir vos fichiers de configuration , qui incluent des mots de passe. Par exemple, si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre serveur est ''d:\biocase\www'' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache (mais sous Ubuntu;, il est placé dans ''/etc/apache2/sites-available'' et lié sous forme de raccourci à ''/etc/apache2/sites-enabled''). Si vous installez BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez dans tous les cas en informer l'administrateur avant de modifier ce fichier, car il s'agît du fichier principal qui règle le comportement du serveur! [[image:instHttpdConf.png|480px]] Sauvez le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utilisez IIS, vous devez créer d'abord un répertoire virtuel pour le chemin physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'). Utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier '''.cgi''' et '''.py''' avec le chemin de votre fichier exécutable Python. Vous devriez ajouter également ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmoins de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'accueil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du type de logiciel de la base de données à publier ainsi que des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'accueil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes). Cette page indique également si cette librairie est présente sur le système et, si dans ce cas, avec quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci: [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe proprement dites sont déjà inclues dans le fichier d'archive que vous avez installé ou bien dans le répertoire Subversion d'où vous êtes parti. La librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos donnes sur le Web). Si vous n'envisagez pas de l'utiliser, il n'est pas nécessaire de l'installer. Mais sinon cliquez sur le lien 'Download', trouvez le fichier d'installation qui convient à votre système d'exploitation, et installez 4SuiteXML à partir du fichier téléchargé. Pour chaque type de base de données (SGBDR ou Système Gestionnaire be base de données Relationnelle dans le jargon), vous devrez toujours installer le package ou driver Python correspondant. Ce driver permet à un programme Python de se connecter à une base de données. Il doit être installé et configuré non pas au niveau de BioCASe, mais au niveau de l'environnement Python. Le BPS peut toutefois vous guider dans leur installation. Si votre collection est par exemple stockée dans une base de données 'MySQL', cliquez sur le lien pour télécharger le module 'MySQL', trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraires dans votre navigateur. La ligne de la liste qui correspond au driver devrait passer de l'orange pour 'non installé' à une indication verte mentionnant la version de la libraire. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utilisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]], actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessaire d'installer Graphviz. Dans le cas contraire nous vous recommandons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquat, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == A la fin de la procédure de configuration, vous devriez changer le mot de passe. L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les paramètres du BPS, dont le mot de passe. Retournez à l'écran d'accueil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la page 'Configuration' dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation. Nous supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''. Vous pouvez encoder un nouveau mot de passe à la quatrièrme ligne de cet écran. Soyez précautionneux en le tapant. Si jamais vous l'oubliez, il est toutefois possible de le réinitialiser en modifiant directement les fichiers de configuration. Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers binaires, car Graphviz n'est pas une librairie Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sur votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary</small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''). Sur un système Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connexion à une base de données), vous devriez vous assurer que tout est installé correctement: # Tapez ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à votre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modifiables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les 'Logs' (fichiers qui enregistrent les opérations sur la machine) et la configuration. Si ce n'est pas le cas, le chemin vers le fichier qui pose problème sera montré. Veuillez vous assurer que Python et/ou Apache peuvent écrire sur ce fichier. Par par exemple sur Ubuntu où Apache correspond à un compte utilisateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre> Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connexion via la librairie ODBC) et que vous ne souhaitez pas utiliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] d7dd8fc605bc5827bdb7b68bdfab6c97bc2587b6 549 548 2012-04-10T16:13:59Z FranckTheeten 15 /* Exécution du script d'installation */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes en fonction de votre degré de familiarité avec Python et Apache. Si vous êtes un débutant, veuillez lire le [[BeginnersGuide_(French)|Guide du Débutant]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit, soit être à l'extérieur d'un pare-feu institutionnel, ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de satisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine vers la base de données à publier, soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL Server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système que vous utilisez. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fonctionner également avec les versions 2.6 et 2.7, la version recommandée est Python 2.5 (en fait il s'agît plutôt de la dernière sous-version stable de la branche 2.5, que nous notons 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le site Web du driver(par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous effectuez l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier ''C:/Program Files'' par défaut. En effet, lorsqu'il est placé à cet endroit, le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur de données BioCASe == Vous pouvez télécharger le BPS soit sous forme d'archive zippée, ou directement à partir du Subversion du BGBM (Subversion ou ''SVN'' est un logiciel qui permet à plusieurs programmeurs de travailler en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la documentation des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion auparavant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (pré-installé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites un clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installation) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau ce script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS P Oar exemple si votre provider est installé à l'emplacement ''D:\\biocase'' tapez <pre>D:</pre><pre>cd D://biocase</pre> Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Ci dessous un exemple sous DOs pour Windows, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python figurant dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de interpréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais pour une machine la production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecture depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur en permettant au monde extérieur de voir vos fichiers de configuration , qui incluent des mots de passe. Par exemple, si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre serveur est ''d:\biocase\www'' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache (mais sous Ubuntu;, il est placé dans ''/etc/apache2/sites-available'' et lié sous forme de raccourci à ''/etc/apache2/sites-enabled''). Si vous installez BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez dans tous les cas en informer l'administrateur avant de modifier ce fichier, car il s'agît du fichier principal qui règle le comportement du serveur! [[image:instHttpdConf.png|480px]] Sauvez le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utilisez IIS, vous devez créer d'abord un répertoire virtuel pour le chemin physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'). Utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier '''.cgi''' et '''.py''' avec le chemin de votre fichier exécutable Python. Vous devriez ajouter également ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmoins de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'accueil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du type de logiciel de la base de données à publier ainsi que des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'accueil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes). Cette page indique également si cette librairie est présente sur le système et, si dans ce cas, avec quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci: [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe proprement dites sont déjà inclues dans le fichier d'archive que vous avez installé ou bien dans le répertoire Subversion d'où vous êtes parti. La librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos donnes sur le Web). Si vous n'envisagez pas de l'utiliser, il n'est pas nécessaire de l'installer. Mais sinon cliquez sur le lien 'Download', trouvez le fichier d'installation qui convient à votre système d'exploitation, et installez 4SuiteXML à partir du fichier téléchargé. Pour chaque type de base de données (SGBDR ou Système Gestionnaire be base de données Relationnelle dans le jargon), vous devrez toujours installer le package ou driver Python correspondant. Ce driver permet à un programme Python de se connecter à une base de données. Il doit être installé et configuré non pas au niveau de BioCASe, mais au niveau de l'environnement Python. Le BPS peut toutefois vous guider dans leur installation. Si votre collection est par exemple stockée dans une base de données 'MySQL', cliquez sur le lien pour télécharger le module 'MySQL', trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraires dans votre navigateur. La ligne de la liste qui correspond au driver devrait passer de l'orange pour 'non installé' à une indication verte mentionnant la version de la libraire. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utilisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]], actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessaire d'installer Graphviz. Dans le cas contraire nous vous recommandons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquat, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == A la fin de la procédure de configuration, vous devriez changer le mot de passe. L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les paramètres du BPS, dont le mot de passe. Retournez à l'écran d'accueil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la page 'Configuration' dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation. Nous supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''. Vous pouvez encoder un nouveau mot de passe à la quatrièrme ligne de cet écran. Soyez précautionneux en le tapant. Si jamais vous l'oubliez, il est toutefois possible de le réinitialiser en modifiant directement les fichiers de configuration. Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers binaires, car Graphviz n'est pas une librairie Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sur votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary</small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''). Sur un système Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connexion à une base de données), vous devriez vous assurer que tout est installé correctement: # Tapez ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à votre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modifiables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les 'Logs' (fichiers qui enregistrent les opérations sur la machine) et la configuration. Si ce n'est pas le cas, le chemin vers le fichier qui pose problème sera montré. Veuillez vous assurer que Python et/ou Apache peuvent écrire sur ce fichier. Par par exemple sur Ubuntu où Apache correspond à un compte utilisateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre> Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connexion via la librairie ODBC) et que vous ne souhaitez pas utiliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] e1b5add66b26696715fcb009315cd16cb7813783 550 549 2012-04-10T16:14:32Z FranckTheeten 15 /* Exécution du script d'installation */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes en fonction de votre degré de familiarité avec Python et Apache. Si vous êtes un débutant, veuillez lire le [[BeginnersGuide_(French)|Guide du Débutant]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit, soit être à l'extérieur d'un pare-feu institutionnel, ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de satisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine vers la base de données à publier, soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL Server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système que vous utilisez. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fonctionner également avec les versions 2.6 et 2.7, la version recommandée est Python 2.5 (en fait il s'agît plutôt de la dernière sous-version stable de la branche 2.5, que nous notons 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le site Web du driver(par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous effectuez l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier ''C:/Program Files'' par défaut. En effet, lorsqu'il est placé à cet endroit, le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur de données BioCASe == Vous pouvez télécharger le BPS soit sous forme d'archive zippée, ou directement à partir du Subversion du BGBM (Subversion ou ''SVN'' est un logiciel qui permet à plusieurs programmeurs de travailler en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la documentation des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion auparavant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (pré-installé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites un clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installation) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau ce script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS P Oar exemple si votre provider est installé à l'emplacement ''D:\\biocase'' tapez <pre>D:</pre><pre>cd D://biocase</pre> Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Ci dessous un exemple sous DOs pour Windows, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python figurant dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de interpréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais, pour une machine en production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecture depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur en permettant au monde extérieur de voir vos fichiers de configuration , qui incluent des mots de passe. Par exemple, si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre serveur est ''d:\biocase\www'' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache (mais sous Ubuntu;, il est placé dans ''/etc/apache2/sites-available'' et lié sous forme de raccourci à ''/etc/apache2/sites-enabled''). Si vous installez BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez dans tous les cas en informer l'administrateur avant de modifier ce fichier, car il s'agît du fichier principal qui règle le comportement du serveur! [[image:instHttpdConf.png|480px]] Sauvez le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utilisez IIS, vous devez créer d'abord un répertoire virtuel pour le chemin physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'). Utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez finalement les extensions de fichier '''.cgi''' et '''.py''' avec le chemin de votre fichier exécutable Python. Vous devriez ajouter également ''index.html'' à la des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmoins de bonnes explications sur le web, par exemple: * http://www.ecp.cc/pyiis.html * http://support.microsoft.com/kb/276494 * http://python.markrowsoft.com/iiswse.asp Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'accueil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du type de logiciel de la base de données à publier ainsi que des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'accueil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes). Cette page indique également si cette librairie est présente sur le système et, si dans ce cas, avec quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci: [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe proprement dites sont déjà inclues dans le fichier d'archive que vous avez installé ou bien dans le répertoire Subversion d'où vous êtes parti. La librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos donnes sur le Web). Si vous n'envisagez pas de l'utiliser, il n'est pas nécessaire de l'installer. Mais sinon cliquez sur le lien 'Download', trouvez le fichier d'installation qui convient à votre système d'exploitation, et installez 4SuiteXML à partir du fichier téléchargé. Pour chaque type de base de données (SGBDR ou Système Gestionnaire be base de données Relationnelle dans le jargon), vous devrez toujours installer le package ou driver Python correspondant. Ce driver permet à un programme Python de se connecter à une base de données. Il doit être installé et configuré non pas au niveau de BioCASe, mais au niveau de l'environnement Python. Le BPS peut toutefois vous guider dans leur installation. Si votre collection est par exemple stockée dans une base de données 'MySQL', cliquez sur le lien pour télécharger le module 'MySQL', trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraires dans votre navigateur. La ligne de la liste qui correspond au driver devrait passer de l'orange pour 'non installé' à une indication verte mentionnant la version de la libraire. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utilisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]], actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessaire d'installer Graphviz. Dans le cas contraire nous vous recommandons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquat, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == A la fin de la procédure de configuration, vous devriez changer le mot de passe. L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les paramètres du BPS, dont le mot de passe. Retournez à l'écran d'accueil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la page 'Configuration' dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation. Nous supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''. Vous pouvez encoder un nouveau mot de passe à la quatrièrme ligne de cet écran. Soyez précautionneux en le tapant. Si jamais vous l'oubliez, il est toutefois possible de le réinitialiser en modifiant directement les fichiers de configuration. Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers binaires, car Graphviz n'est pas une librairie Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sur votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary</small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''). Sur un système Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connexion à une base de données), vous devriez vous assurer que tout est installé correctement: # Tapez ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à votre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modifiables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les 'Logs' (fichiers qui enregistrent les opérations sur la machine) et la configuration. Si ce n'est pas le cas, le chemin vers le fichier qui pose problème sera montré. Veuillez vous assurer que Python et/ou Apache peuvent écrire sur ce fichier. Par par exemple sur Ubuntu où Apache correspond à un compte utilisateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre> Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connexion via la librairie ODBC) et que vous ne souhaitez pas utiliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] 98e565fd16c9235d1c13e9fb09dd5fafda763a5f 555 550 2012-04-12T14:03:41Z FranckTheeten 15 wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes en fonction de votre degré de familiarité avec Python et Apache. Si vous êtes un débutant, veuillez lire le [[BeginnersGuide_(French)|Guide du Débutant]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit, soit être à l'extérieur d'un pare-feu institutionnel, ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de satisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine vers la base de données à publier, soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL Server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système que vous utilisez. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fonctionner également avec les versions 2.6 et 2.7, la version recommandée est Python 2.5 (en fait il s'agît plutôt de la dernière sous-version stable de la branche 2.5, que nous notons 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le site Web du driver(par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous effectuez l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier ''C:/Program Files'' par défaut. En effet, lorsqu'il est placé à cet endroit, le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur de données BioCASe == Vous pouvez télécharger le BPS soit sous forme d'archive zippée, ou directement à partir du Subversion du BGBM (Subversion ou ''SVN'' est un logiciel qui permet à plusieurs programmeurs de travailler en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la documentation des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion auparavant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (pré-installé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites un clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installation) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau ce script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS P Oar exemple si votre provider est installé à l'emplacement ''D:\\biocase'' tapez <pre>D:</pre><pre>cd D://biocase</pre> Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Ci dessous un exemple sous DOs pour Windows, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python figurant dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de interpréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais, pour une machine en production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecture depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur en permettant au monde extérieur de voir vos fichiers de configuration , qui incluent des mots de passe. Par exemple, si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre serveur est ''d:\biocase\www'' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache (mais sous Ubuntu;, il est placé dans ''/etc/apache2/sites-available'' et lié sous forme de raccourci à ''/etc/apache2/sites-enabled''). Si vous installez BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez dans tous les cas en informer l'administrateur avant de modifier ce fichier, car il s'agît du fichier principal qui règle le comportement du serveur! [[image:instHttpdConf.png|480px]] Sauvez le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utilisez IIS, vous devez créer d'abord un répertoire virtuel pour le chemin physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'). Vous devriez ajouter également ''index.html'' à la liste des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Pour associer les scripts Python avec Python dans les versions 5 et 6 d'IIS, utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez les extensions de fichier '''.cgi''' et '''.py''' avec le chemin de votre fichier exécutable Python (Voyez la copie d'écran ci-dessous). [[File:instAddScriptMap.png|330px]] Avec la version 7, vous pouvez ouvrir cette fenêtre en allant sur <small>Add Script Map</small>. Notez les "%s%s" après l'emplacement du Python, qui indiquent des emplacement réservés pour les paramètres. Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmoins de bonnes explications sur le web, par exemple: * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'accueil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du type de logiciel de la base de données à publier ainsi que des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'accueil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes). Cette page indique également si cette librairie est présente sur le système et, si dans ce cas, avec quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci: [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe proprement dites sont déjà inclues dans le fichier d'archive que vous avez installé ou bien dans le répertoire Subversion d'où vous êtes parti. La librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos donnes sur le Web). Si vous n'envisagez pas de l'utiliser, il n'est pas nécessaire de l'installer. Mais sinon cliquez sur le lien 'Download', trouvez le fichier d'installation qui convient à votre système d'exploitation, et installez 4SuiteXML à partir du fichier téléchargé. Pour chaque type de base de données (SGBDR ou Système Gestionnaire be base de données Relationnelle dans le jargon), vous devrez toujours installer le package ou driver Python correspondant. Ce driver permet à un programme Python de se connecter à une base de données. Il doit être installé et configuré non pas au niveau de BioCASe, mais au niveau de l'environnement Python. Le BPS peut toutefois vous guider dans leur installation. Si votre collection est par exemple stockée dans une base de données 'MySQL', cliquez sur le lien pour télécharger le module 'MySQL', trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraires dans votre navigateur. La ligne de la liste qui correspond au driver devrait passer de l'orange pour 'non installé' à une indication verte mentionnant la version de la libraire. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utilisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]], actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessaire d'installer Graphviz. Dans le cas contraire nous vous recommandons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquat, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == A la fin de la procédure de configuration, vous devriez changer le mot de passe. L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les paramètres du BPS, dont le mot de passe. Retournez à l'écran d'accueil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la page 'Configuration' dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation. Nous supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''. Vous pouvez encoder un nouveau mot de passe à la quatrièrme ligne de cet écran. Soyez précautionneux en le tapant. Si jamais vous l'oubliez, il est toutefois possible de le réinitialiser en modifiant directement les fichiers de configuration. Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers binaires, car Graphviz n'est pas une librairie Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sur votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary</small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''). Sur un système Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connexion à une base de données), vous devriez vous assurer que tout est installé correctement: # Tapez ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à votre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modifiables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les 'Logs' (fichiers qui enregistrent les opérations sur la machine) et la configuration. Si ce n'est pas le cas, le chemin vers le fichier qui pose problème sera montré. Veuillez vous assurer que Python et/ou Apache peuvent écrire sur ce fichier. Par par exemple sur Ubuntu où Apache correspond à un compte utilisateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre> Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connexion via la librairie ODBC) et que vous ne souhaitez pas utiliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] ac69d49b31bb1a7cfd57c89fc4ec2be054f41797 556 555 2012-04-12T14:04:49Z FranckTheeten 15 /* Internet Information Server (IIS) */ wikitext text/x-wiki '''Traduction: Franck Theeten''' '''Musée Royal de l'Afrique Centrale de Tervuren''' '''Pour le projet CABIN (Central African Biodiversity Information Network)''' Cette partie du Wiki explique comment installer le logiciel fournisseur de données BioCASe (‘BioCASe Provider Software’ aussi appelé "BPS"). Cela devrait prendre entre 15 et 60 minutes en fonction de votre degré de familiarité avec Python et Apache. Si vous êtes un débutant, veuillez lire le [[BeginnersGuide_(French)|Guide du Débutant]] en premier lieu. == Prérequis == Avant de commencer l’installation, assurez-vous que le système sur lequel vous avez l'intention d'installer le BPS remplit les conditions suivantes: # Vous avez des privilèges d'administrateur sur cette machine, c'est à dire les droits d'utilisateur pour installer de nouveaux logiciels. # Il est accessible de l'extérieur de votre établissement. Cela signifie qu'il doit, soit être à l'extérieur d'un pare-feu institutionnel, ou que, plus probablement, le port HTTP pour un accès Web est ouvert pour un pare-feu existant [note : il n’est pas nécessaire de satisfaire cette condition si vous installez BioCASe a des fins de test ou sur votre propre ordinateur pour vous entraîner individuellement]. # Une connexion directe peut être établie à partir de cette machine vers la base de données à publier, soit parce qu'il n'y a pas de pare-feu entre les deux, ou bien que le port utilisé par le SGBD est ouvert (par exemple 3306 pour MySQL, 5432 pour PostGres, 1433 pour Microsoft SQL Server). Si le logiciel BioCASe est installé sur la même machine que le serveur de base de données, le pare-feu n’a normalement pas à être ouvert. Si une de ces conditions n'est pas remplie, veuillez contacter votre administrateur système pour de l'aide, lisez la [[FAQ]] ou [[Special:Contact|prenez contact avec l'équipe BioCASe]]. == Installation de Apache / IIS et Python == Si vous utilisez un système Linux, vous n'avez probablement pas besoin d'installer ces logiciels, car Apache et Python sont généralement préinstallés. Il en va de même pour MacOS X, mais vous devriez malgré tout jeter un oeil aux discussions et à la documentations accessible sur le web pour obtenir plus d'information sur les emplacements de fichiers dans le système que vous utilisez. Même si le provider fonctionne avec Apache et Internet Information Server (IIS) de Microsoft , nous vous recommandons d'utiliser Apache, qui est beaucoup plus facile à configurer. Toutefois, si vous utilisez déjà IIS, vous souhaitez peut-être le conserver. Les versions d'Apache recommandées sont 2.0 et 2.2. Le BPS nécessite la version 2.5 de Python afin de fonctionner. Même s’il peut fonctionner également avec les versions 2.6 et 2.7, la version recommandée est Python 2.5 (en fait il s'agît plutôt de la dernière sous-version stable de la branche 2.5, que nous notons 2.5+). Pour les versions ultérieures, les librairies nécessaires pour la connexion du BPS à la base de données utilisé ne sont peut être pas encore disponible. Vous pouvez aller à la [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi page de test des librairies de l'installation du Musée et Jardin Botanique de Berlin] pour vous procurer le paquet d'installation du driver de votre base de données, et vérifier sa disponibilité pour de nouvelles versions de Python sur le site Web du driver(par exemple, la bibliothèque 4SuiteXML utilisée pour l'exécution de l’interface de requête locale est actuellement uniquement disponibles pour Python 2.5). Soit dit en passant: vous pouvez installer plusieurs versions différentes de Python sur la même machine sans conflits. Si vous effectuez l’installation sur un ordinateur Windows Vista ou Windows 7, nous vous recommandons de ne pas installer Apache dans le dossier ''C:/Program Files'' par défaut. En effet, lorsqu'il est placé à cet endroit, le serveur Apache a un comportement imprévisible qui peut vous causer des ennuis lors de la modification de fichiers de configuration. Il suffit de choisir un autre chemin (par exemple C:\\Apache22 ou D:\\Apache22) pour parer à cet inconvénient. Dès que vous avez installé avec succès Apache et Python, vous pouvez procéder à l'étape suivante. == Téléchargement du logiciel fournisseur de données BioCASe == Vous pouvez télécharger le BPS soit sous forme d'archive zippée, ou directement à partir du Subversion du BGBM (Subversion ou ''SVN'' est un logiciel qui permet à plusieurs programmeurs de travailler en parallèles sur le même code, il facilite aussi l'archivage des versions du code et la documentation des changements en les stockant sur un réseau). La première option est la plus facile si vous n'êtes pas familier avec Subversion. Avec la deuxième option, il sera plus facile de mettre à jour votre installation ultérieurement. Si vous n'avez jamais utilisé Subversion auparavant et ne prévoyez pas de mise à jour régulière de votre installation, choisissez l'archive. Mais si vous prévoyez de mettre en place une installation de production qui devrait être mise à jour, vous devriez opter plutôt pour le dépôt Subversion. === Téléchargement du fichier d'archive === Il suffit de télécharger l'archive à partir de la partie [http://www.biocase.org/products/provider_software Téléchargement] du site BioCASe et de décompresser le contenu du dossier de destination. === Téléchargement à partir du dépôt Subversion === Si vous utilisez Windows, vous devez installer un client Subversion premier (vous pouvez essayer [http://tortoisesvn.net Tortoise], qui est livré avec une jolie interface très commode). Linux et MacOS contiennent habituellement un client Subversion préinstallé. L'URL de la dernière version stable est http://ww2.biocase.org/svn/bps2/branches/stable, que nous vous recommandons d'utiliser. La mise à jour de votre installation avec la dernière version BPS est alors facile, et revient à exécuter la commande « update » de Subversion. Si vous voulez télécharger une version définie, l'URL serait (exemple dans le cas de la version 2.6.0) http://ww2.biocase.org/svn/bps2/tags/release_2.6.0 . Pour télécharger le BPS avec le client Subversion en ligne de commande (pré-installé sur Linux / MacOS), allez simplement au répertoire sur lequel vous voulez décompressez l’archive et introduisez : <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre> Vous vous connectez alors au serveur du Jardin et Musée Botanique de Berlin avec un accès anonyme et télécharger tous les fichiers du sous-répertoire ''biocase''. Dans le cas où vous utilisez Tortoise sous Windows, faites un clic-droit sur le répertoire de destination dans l'explorateur Windows et choisissez «<small>Checkout</small>». Introduisez l'URL (http://ww2.biocase.org/svn/bps2/branches/stable), changez le nom du dossier de destination sur votre machine pour qu’il soit significatif et facilement reconnaissable par vous et appuyez sur OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] Nous supposerons dans la suite de cette explication que vous avez installé le logiciel BPS dans un dossier nommé BioCASe, qui devrait ressembler à ceci maintenant: [[image:instBiocaseFolder.png|480px]] == Exécution du script d'installation == Le script d'installation va paramétrer les fichiers de configuration avec les valeurs que vous lui fournissez (en fonction de votre installation) et lier les fichiers de script à votre installation de Python. Vous pouvez ré-exécuter le script de configuration dans le cas où vous modifiez l’emplacement d’Apache, Python ou du logiciel BioCASe lui-même sur votre ordinateur. Si vous souhaitez modifier ou corriger l'un des paramètres que vous avez déclarés pendant l'installation, vous pouvez également changer les valeurs en relançant à nouveau ce script d’installation. Pour exécuter le script, ouvrez une fenêtre de terminal (appelée «invite de commande", "commandes MS-DOS" ou encore "command prompt" en anglais dans Windows) à laquelle vous pouvez accéder sur Windows via le sous-menu « accessoire » du menu général, ou bien en tapant «cmd» dans le menu « exécuter ». Changez le répertoire de travail vers le dossier que vous avez téléchargé le BPS P Oar exemple si votre provider est installé à l'emplacement ''D:\\biocase'' tapez <pre>D:</pre><pre>cd D://biocase</pre> Tapez ensuite:<pre>python setup.py</pre> Si la commande ne marche pas, vous devez peut-être ajouter le nom de chemin complet vers l'exécutable Python pour que la fenêtre d'installation puisse le trouver. Ci dessous un exemple sous DOs pour Windows, en supposant que vous avez installé Python dans sa version 2.5 et dans le répertoire C:\Python25: <pre>C:\Python25\python.exe setup.py</pre> ...adaptez les valeurs en fonction de la situation sur votre ordinateur) Assurez-vous que la version de Python figurant dans la cinquième ligne de la sortie correspond à celle que vous désirez (sinon ajoutez le chemin complet vers la bonne version de interpréteur comme montré ci-dessous): [[image:instSetupScriptStart.png|480px]] Les questions posées par le script de configuration proposent des valeurs par défaut. Si vous souhaitez les conserver, il suffit d'appuyer sur <small>Entrée</small>. <strong>Webserver domain:</strong> Il s'agit du nom de domaine du serveur Web que vous installez sur BioCASE, par exemple, '''http://www.moninstitution.org'''. Vous pouvez conserver la valeur par défaut ''localhost'' tant que vous configurez et testez l'installation localement, mais, pour une machine en production réellement accessible par Internet, vous aurez besoin de changer cela dans le nom de domaine correct (ou adresse IP) ultérieurement. <strong>Base URL:</strong> Ce sera la deuxième partie de l'URL de votre installation (l’adresse du site dans le serveur, par exemple ''http://moninstitution.org/monbiocase/'' ). Si vous gardez la valeur par défaut, votre installation sera accessible plus tard à ''www.myinstitution.org/biocase''. Ce paramètre correspond à une option des paramètres Apache. Après avoir répondu à ces deux questions, le script mettra à jour vos fichiers. À la fin de la sortie que vous pouvez trouver les lignes dont vous aurez besoin plus tard pour créer le lien entre BioCASe et votre serveur web. Il ne faut donc pas fermer la fenêtre! [[image:instSetupScriptDone.png|480px]] == Configurer votre serveur Web pour BioCASe == Les fichiers et programmes visibles publiquement depuis le Web sont placés dans le sous-répertoire '''www''' de votre installation BioCASe et doivent être configurés sur votre serveur pour un accès en lecture depuis l'extérieur. Vous devriez toujours configurer cela à l'aide d'un répertoire virtuel: c'est à dire que vous ne devez pas installer BioCASe à la racine de votre serveur Web (par exemple sur '/var/www' sur Ubuntu). Sinon vous pouvez compromettre la confidentialité de votre serveur en permettant au monde extérieur de voir vos fichiers de configuration , qui incluent des mots de passe. Par exemple, si vous avez installé BioCASe sur ''d:\biocase'', le répertoire que vous devez déclarer comme répertoire virtuel dans la configuration de votre serveur est ''d:\biocase\www'' et sera accessible par Internet via ''http://www.moninstitution.org/biocase''. === Apache === Si vous utilisez Apache, l'ajout du répertoire BioCASe dans votre environnement se fait simplement en copiant les directives affichées à la fin du script d'installation dans le fichier de configuration d'Apache (''httpd.conf'' dans le cas de la plupart des installations sur Windows). Ce fichier est le plus souvent situé dans le sous-répertoire ''conf'' de votre installation d'Apache (mais sous Ubuntu;, il est placé dans ''/etc/apache2/sites-available'' et lié sous forme de raccourci à ''/etc/apache2/sites-enabled''). Si vous installez BioCASe sur un serveur qui est déjà utilisé pour publier des sites webs, veuillez dans tous les cas en informer l'administrateur avant de modifier ce fichier, car il s'agît du fichier principal qui règle le comportement du serveur! [[image:instHttpdConf.png|480px]] Sauvez le fichier et redémarrez Apache pour mettre en vigueur les modifications (sous Windows ce redémarrage est souvent possible via l'interface graphique de votre Apache ou du package WAMP que vous avez installés, sous Linux via une commande Shell, comme sous Ubuntu<pre>sudo /etc/init.d/apache2 force-reload</pre> . === Internet Information Server (IIS) === Si vous utilisez IIS, vous devez créer d'abord un répertoire virtuel pour le chemin physique ''c:\biocase\www'' en utilisant l'outil ''IIS Server Manager''. Allouez tous les droits pour ce répertoire ('Read', 'Write', 'Script Execution', 'Directory listing'). Vous devriez ajouter également ''index.html'' à la liste des documents par défaut (<small>Virtual Directory > Properties > Documents > Default</small>). Pour associer les scripts Python avec Python dans les versions 5 et 6 d'IIS, utilisez le bouton <small>Configure</small> sur l'onglet ''virtual directory'' et associez les extensions de fichier '''.cgi''' et '''.py''' avec le chemin de votre fichier exécutable Python (Voyez la copie d'écran ci-dessous). [[File:instAddScriptMap.png|330px]] Avec la version 7, vous pouvez ouvrir cette fenêtre en allant sur <small>Add Script Map</small>. Notez les "%s%s" après l'emplacement du Python, qui indiquent des emplacement réservés pour les paramètres. Nous savons que configurer d'une manière générale Python pour IIS est souvent fastidieux. On trouve néanmoins de bonnes explications sur le web, par exemple: * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html === Testez le lien === Dès que vous avez terminé, vous pouvez tester votre installation de BioCASe en ouvrant un navigateur et en allant sur ''http://localhost/biocase'' (replacez ''localhost'' par le domaine sur laquelle vous avez installé BioCASe si vous ne travaillez pas localement). Vous devriez alors voir la page d'accueil du BPS: [[image:instBiocaseStartpage.png|500px]] == Installer les paquets additionnels Python == Avant de pouvoir configurer votre installation, vous devez installer des extensions, en fonction du type de logiciel de la base de données à publier ainsi que des fonctionnalités du BPS que vous souhaitez utiliser. Sur la page d'accueil du BPS, cliquez sur <small>Utilities</small>, puis <small>library test</small> pour naviguer sur la page permettant de tester la librairie (paquet de programmes). Cette page indique également si cette librairie est présente sur le système et, si dans ce cas, avec quelle version. Sur une installation neuve, vous devriez voir un écran similaire à celui-ci: [[image:instLibrariesVanilla.png|450px]] Les librairies spécifiques à BioCASe proprement dites sont déjà inclues dans le fichier d'archive que vous avez installé ou bien dans le répertoire Subversion d'où vous êtes parti. La librairie '4SuiteXML' est le nécessaire pour Local Query Tool (une interface de recherche pré-configurées qui vous permet d'accéder à vos donnes sur le Web). Si vous n'envisagez pas de l'utiliser, il n'est pas nécessaire de l'installer. Mais sinon cliquez sur le lien 'Download', trouvez le fichier d'installation qui convient à votre système d'exploitation, et installez 4SuiteXML à partir du fichier téléchargé. Pour chaque type de base de données (SGBDR ou Système Gestionnaire be base de données Relationnelle dans le jargon), vous devrez toujours installer le package ou driver Python correspondant. Ce driver permet à un programme Python de se connecter à une base de données. Il doit être installé et configuré non pas au niveau de BioCASe, mais au niveau de l'environnement Python. Le BPS peut toutefois vous guider dans leur installation. Si votre collection est par exemple stockée dans une base de données 'MySQL', cliquez sur le lien pour télécharger le module 'MySQL', trouvez le fichier d'installation qui correspond à votre version de Python et à votre système d'exploitation et faites l'installation. Une fois l'installation achevée, rechargez la page de test des libraires dans votre navigateur. La ligne de la liste qui correspond au driver devrait passer de l'orange pour 'non installé' à une indication verte mentionnant la version de la libraire. Répétez cette opérations si vous devez connecter d'autres types de base de données. La librairie "Graphviz" figurant parmi les librairies extérieures optionnelles est utilisée par le BPS pour créer une représentation graphique du modèle de donénes de votre bases de données dans la partie du site web utilisée pour enregistrer les tables (plus d'information sur cela dans [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]], actuellement en anglais). Si votre modèle de données est très simple (c'est à dire, si la structure de votre BDD ne contient pas plus de 3 ou 4 tables), il n'est sans doute pas nécessaire d'installer Graphviz. Dans le cas contraire nous vous recommandons l'installation : cliquez simplement sur le lien 'Download link', trouvez le fichier d'installation adéquat, et faites comme mentionné plus haut pour les autres librairies. == Changer le mot de passe par défaut == A la fin de la procédure de configuration, vous devriez changer le mot de passe. L'outil de configuration ('configuration tool' en anglais) est une interface web qui vous permet de configurer presque tous les paramètres du BPS, dont le mot de passe. Retournez à l'écran d'accueil du BPS (cliquez sur <small>Start</small> au sommet de la page, ou allez sur ''http://localhost/biocase'' si le serveur est sur votre machine), choisissez <small>Config Tool</small>, cliquez alors sur <small>System Administration</small>. L'interface vous demandera alors un mot de passe. Tapez le mot de passe par défaut ''ACDC'', cliquez finalement sur <small>Submit</small>. Vous serez alors redirigés vers la page 'Configuration' dont la partie supérieure devrait ressembler à: [[image:instConfigTool.png|500px]] Dans les deux premières lignes, vous verrez les valeurs que vous avez entrer en exécutant le fichier d'installation. Nous supposons ici que vous avez laissé les valeurs par défaut ''localhost'' et ''biocase''. Vous pouvez encoder un nouveau mot de passe à la quatrièrme ligne de cet écran. Soyez précautionneux en le tapant. Si jamais vous l'oubliez, il est toutefois possible de le réinitialiser en modifiant directement les fichiers de configuration. Appuyez sur <small>Update config</small> pour sauvegarder le nouveau mot de passe dans les fichiers de configuration. Si vous avez installé Graphviz, vous devrez préciser au BPS l'emplacement des fichiers binaires, car Graphviz n'est pas une librairie Python enregistrée directement au niveau de la configuration globale de Python. Pour ce faire, localisez l'emplacement de Graphviz sur votre disque dur et entrez le chemin de l'exécutable dans la zone de texte marquée par <small>Graphviz dot binary</small>. Sur des machines Linux elle devrait être similaire à l'emplacement par défaut (''/usr/local/bin/dot''). Sur un système Windows ce sera probablement quelque chose comme ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Pressez <small>update config</small> pour sauver ce chemin. == Vérifiez votre Installation == Avant de passer à l'étape suivante (configuration d'une connexion à une base de données), vous devriez vous assurer que tout est installé correctement: # Tapez ''http://localhost/biocase'' (ou bien le nom de domaine et l'adresse qui correspond à votre installation) vous mène à la page d'acceuil du logiciel BioCASe (comme décrit à la section [[#Configurer_votre_serveur_Web_pour_BioCASe|#Configurer votre serveur Web pour BioCASe]]). # La page de test des librairies (<small>Utilities > library test</small>) mentionne que vous avez installé les paquets ou drivers pour votre BDD, 4SuiteXML (si vous souhaitez utiliser le Local Query Tool)et Graphviz Dot (optionnel, voir plis haut). # <small>Status of writeable directories</small> ('Statut des répertoires modifiables') montre <small>writeable</small> ('inscriptible') en vert à la fois pour les 'Logs' (fichiers qui enregistrent les opérations sur la machine) et la configuration. Si ce n'est pas le cas, le chemin vers le fichier qui pose problème sera montré. Veuillez vous assurer que Python et/ou Apache peuvent écrire sur ce fichier. Par par exemple sur Ubuntu où Apache correspond à un compte utilisateur appelé 'www-data' <pre>sudo chown -R www-data:www-data /var/www/biocase/log /var/www/biocase/config</pre><pre>sudo chmod -R 755 /var/www/biocase/log /var/www/biocase/config</pre> Supposons, par exemple, que vous avez choisi d'utiliser BioCASe en combinaison avec deux bases de données, l'une en PostgreSQL et l'autre en Microsoft Access (pour laquelle il faut une connexion via la librairie ODBC) et que vous ne souhaitez pas utiliser le Query Tool, la page de vérification des librairies devrait alors ressembler à cela: [[image:instLibrariesDone.png|450px]] b9e02fdb4de393ad428c879a8e618b1cffa073c8 0 23 543 499 2012-04-10T16:08:54Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Fixes for XML archives: ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ===Version 3.0.1 [2012-04-05]=== * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for XML archives when using IIS (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 6f193524145ef951f94d73493fc60bce12f57f37 557 543 2012-04-16T16:07:40Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Fixes for XML archives: ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Archiving binary moved from archive working folder to archive lib folder ===Version 3.0.1 [2012-04-05]=== * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for XML archives when using IIS (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool a4b190d65c0010f156fe0e3857bc31104e274a54 558 557 2012-04-19T13:40:42Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Version 3.1 (Upcoming) === * Fixes for XML archives: ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Archiving binary moved from archive working folder to archive lib folder ** Compatibility issued with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 32dd60270bb018a31bcec8327f8f49bb23b7e9dd 559 558 2012-04-19T13:41:26Z JoergHoletschek 3 /* Version 3.1 (Upcoming) */ wikitext text/x-wiki == Current Version 3 == === Version 3.1 (Upcoming) === * Fixes for XML archives: ** Compatibility issued with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 7878a9054859e98510e560c94d6cf910b56346ff 560 559 2012-04-19T15:52:58Z JoergHoletschek 3 /* Version 3.1 (Upcoming) */ wikitext text/x-wiki == Current Version 3 == === Version 3.1 (Upcoming) === * Fixes for XML archives: ** Compatibility issued with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 35c9b708580e32d8b3ff6e9fb39b62fb7b180570 561 560 2012-04-23T13:53:30Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issued with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 1cff7c40947e7c5659010d282ea569320fec543f 562 561 2012-04-23T13:53:47Z JoergHoletschek 3 /* Version 3.1 [2012-04-23] */ wikitext text/x-wiki == Current Version 3 == === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 6aa5032cb3139af437f8652cced6244b1ca977ea 563 562 2012-05-09T12:11:10Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Version 3.2 [upcoming] === * Fixes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool f8758408155c0440b3b70c116177e43ef9f5bc81 564 563 2012-05-31T12:59:55Z JoergHoletschek 3 /* Version 3.2 [upcoming] */ wikitext text/x-wiki == Current Version 3 == === Version 3.2 [upcoming] === * Fix for page 'DB Structure': DB graph generation works now also when IIS is used * Fixes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 9892500d5d01fc589813cb21b0543b413041d7fd 565 564 2012-05-31T13:00:51Z JoergHoletschek 3 /* Version 3.2 [upcoming] */ wikitext text/x-wiki == Current Version 3 == === Version 3.2 [upcoming] === * Fix for page 'DB Structure': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Fixes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 7a7e177e3a7b125ffa4f9ce3dd19440eecfeb89c 566 565 2012-05-31T13:01:57Z JoergHoletschek 3 /* Version 3.2 [upcoming] */ wikitext text/x-wiki == Current Version 3 == === Version 3.2 [upcoming] === * Fix for page 'DB Structure': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Fixes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool a724769c20a8ea882c8f058d4447b8829f72a518 567 566 2012-05-31T13:02:14Z JoergHoletschek 3 /* Version 3.2 [upcoming] */ wikitext text/x-wiki == Current Version 3 == === Version 3.2 [upcoming] === * Fix for page 'DB Structure': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 5fac2ad43caaab5c2441f4eb76ee84d7ff43a332 568 567 2012-05-31T13:02:35Z JoergHoletschek 3 /* Version 3.2 [upcoming] */ wikitext text/x-wiki == Current Version 3 == === Version 3.2 [upcoming] === * Fixes for page 'DB Structure' ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Changes for XML archiving ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool d4e2a0196f641ee316aa925f00b4a55dd1a97371 569 568 2012-05-31T13:02:52Z JoergHoletschek 3 /* Version 3.2 [upcoming] */ wikitext text/x-wiki == Current Version 3 == === Version 3.2 [upcoming] === * Fixes for page 'DB Structure': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool aa94473d0961c72bdaf9a3abc15159efe928a14a 0 98 553 2012-04-11T11:14:04Z JoergHoletschek 3 Created page with "This tutorial explains in detail how to set up a datasource. == A BioCASe Data Source == Publishing a database with BioCASe means it will be made accessible through a BioCASe we..." wikitext text/x-wiki This tutorial explains in detail how to set up a datasource. == A BioCASe Data Source == Publishing a database with BioCASe means it will be made accessible through a BioCASe web service, which will be used later by the biodiversity network(s) you want to join to access to your data. For example, the GBIF Harvester will send a BioCASe Scan request to that web service to get a list of all scientific names kept in the database, and subsequently several BioCASe Search requests to load all records for these names. For each database you want to publish, you need to set up a BioCASe data source (do not mix that up with ODBC data sources on Windows machines). The resulting BioCASe web service is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source. So if you made your installation available at http://www.foobar.org/biocase during the installation process and set up a data source named Herbar, the URL of the BioCASe web service would be http://www.foobar.org/biocase/pywrapper.cgi?dsa=Herbar (data source names can be case sensitive, depending on your server’s operating system). == Creating a New Data Source == The start page of the BPS will always list the data sources currently configured, that means the web services offered by that BioCASe installation. On a fresh installation, it will be empty: [[File:instBiocaseStartpage.png|500px]] Creating a new data source can be done in the configuration tool. Choose <small>Config Tool</small> on the start page, then <small>System Administration</small>. Log in with the password you set during the installation process (if you didn’t change that yet, the default is ''ACDC''). In the configuration screen, the sections <small>Data Sources</small> and <small>Recycle Bin</small> can be used to create, remove und restore a data source: [[File:dsaCreateDatasourceEmpty.png|500px|border]] For now, we want to create a new, empty data source. Enter a name – without any spaces – in the text box and click <small>Create DSA</small> (remember that the datasource name is case sensitive on Linux machines). After a second, it will appear in the list of existing data sources with a clickable link. By doing this, you created the first BioCASe web service for your installation. Clicking <small>Remove</small> will uninstall this service again (putting it in the recycle bin, so you can restore it later). Tip: If you select <small>abcdmetadata</small> in the template drop-down list, your data source will be created with an ABCD mapping for all the metadata elements stored in the metadata table mentioned in the [[Preparation]] guide. So if you did use the templates in ''sampledb/abcdmetadata'' to create the metadata table in your database, you can use this template to spare the hassle of mapping this table and start right away with your occurrence data. == Setting up the database connection == Clicking on the data source you’ve just created will take you to the datasource configuration tool. Note that the <small>Schemas</small> section will be empty if you’ve created an empty data source or list two schemas (ABCD 1.2 and 2.06) if you created the datasource based on the <small>abcdmetadata</small> template: [[File:dsaOverview.png|450px]] The link in the <small>Database Connection</small> section will take you to the connection parameters dialog. Fill in the values for your database, press <small>Save</small> and the connection status should turn into a nice green <small>OK</small>. '''DBMS:''' Select the entry appropriate for your collection. The BPS can directly connect to database servers (MySQL, Postgres, DB2, SQL Server, and Oracle) and to Desktop-DBMS like Access and FoxPro through ODBC. ODBC is only available on Windows, so if you plan to use it, BioCASe must be installed on a Windows Server. '''Host:''' Enter the IP address of the database server. If you’re connecting through ODBC, you must leave this empty. Instead, use Windows’ ODBC Administrator to set up a System (!) datasource; the server or file name of the database will be stored in the ODBC configuration. '''Database:''' Enter the name of the ODBC datasource if you’re using ODBC, or the name of the database if you’re directly connecting to a server. '''User/Password:''' Database credentials to be used. As written in the [[Preparation]] guide, you should create a user with just read access for this. '''Encoding:''' The encoding of your database. If you’re not sure, leave the default, you can change it if you run into problems later. If you know your database is using a different encoding than latin-1 (which is typically the default for western languages), maybe UTF-8, select the appropriate entry in the drop-down list. [[File:dsaDBConnection.png|450px]] Once you’ve entered the correct values, press <small>Save</small>. Click <small>Overview</small> in the link list at the top of the page to return to the datasource configuration page. If the connection status doesn’t turn into <small>OK</small> (and doesn’t after you’ve returned to the overview page), please read the [[Debugging]] guide about how to trace the problem. == Setting up the Database Structure == Clicking on <small>Edit DB Structure</small> on the datasource overview page will show the DB structure setup. Make sure the connection status at the top of the page still is <small>OK</small>. In a first step, you will add all tables/views that hold information you want to publish with BioCASe. You can ignore all database tables storing only data you don’t plan to map to ABCD. Same holds true if you use cache tables (as explained in the [[Preparation]] guide): Only the cache tables need to be set up on the database structure page. In a second step, you will specify the foreign key references between these tables. === Adding Tables/Views with their Primary Keys === To add a table or view, just select it in the drop-down list. The name of the table will be filled in the <small>Alias</small> text box (you will use this alias later in the mapping process), feel free to change it into something that describes the contents of that tables (for example ''specimen'' for the root specimen table). Press <small>Add</small> to insert the table into the list. Before continuing to the next table, select the primary key in the list of attributes under <small>Primary Key Attribute(s)</small>. In case the data type is a string and not an integer, select <small>text</small> from the drop-down list. Press <small>Add</small> to add the primary key. If the primary key consists of several columns, you can add several primary key attributes. '''<u>Hint</u>''' It has been noted that diacritic characters in primary key columns can cause problems, so you should try to use integer or ascii text columns as primary keys. Redo these steps (adding the tables first, then the primary key/keys) for all tables you want to publish data from. Always remember to set the correct primary key column(s), even if they’re not defined as primary keys in the database. '''<u>Important Note:</u>''' Tables without a primary key cannot be used by the Provider Software and will be removed upon saving!''' === Specifying Foreign Key References === Once you’ve added all tables, you can define the relations between these tables by defining foreign keys. In order to do this, go to the <small>Foreign Key(s)</small> column of the table that holds a foreign key reference to another table, select the alias of the referenced table and press <small>Add</small>. Select the attribute that stores the foreign key and change the data type if it is not integer, then press <small>Add</small>. You must specify as many attributes for the foreign keys as the primary key of the referenced table has, otherwise the Provider Software won’t be able to create a valid join between the tables. Also, they must be in the same order and have identical data types. Redo this for all tables until you’ve specified all foreign key references. If you’ve installed the Graphviz package, the Provider Software will create a graph visualizing the current database structure. Each table will be visible as a node with the primary key(s) listed. References between tables will be represented by arrows, the foreign key(s) listed in the referring table: [[File:dsaDatabaseStructure_AlgenEngels.png|800px]] === Saving, Editing an Existing Setup === You can change aliases of tables you’ve already added to the list. To do so, go to the <small>Alias</small> text box of that table and type in a different name. Upon exiting the text box, all references to this alias will be updated in the table setup; also the graph will reflect the changes. But caution: Schema mappings use these aliases to bind schema elements to tables and columns, if you change an alias you’ve already used in a schema mapping, you will break this binding! So it is advisable to decide for meaningful alias names and keep them stable once you’ve started the mapping process. Once you’ve added a table/view to the table list, you can’t change the underlying table, since this would invalidate all the primary and foreign key attributes. If you’ve added the wrong table, remove it by using the trash can symbol next to <small>Delete Alias</small> and add the correct table/view. If you’ve finished the table setup and the graph displays the model of your database correctly, save the configuration by pressing <small>Save</small> at the bottom of the page. This is important; navigating away from the page without saving will cause your changes to be lost (even though you will be asked for confirmation)! It is advisable to save the table setup every now and then, for example once you’ve added all tables and primary keys. If you edit an existing table setup and happen to realize you’ve messed it up, you can revert to the state when you’ve pressed <small>Save</small> for the last time by using the <small>Revert</small> button at the bottom. But be careful: All unsaved changes will be lost. '''<u>Important Note:</u>''' The retrieval of tables/view/column names is currently only supported for the following DBMS: Microsoft SQL Server, MySQL, Oracle, Access, Postgres and FoxPro. If you use a different DBMS, you will see a simple text box instead of a drop-down-list, where you’ll have to type in the name of the table or column you want to add. d058710ab56d5bb600216cedcaf7ca7431074ea5 0 99 554 2012-04-11T11:14:56Z JoergHoletschek 3 Created page with "This tutorial will explain how to set up a mapping for the ABCD2.06 data schema. If you intend to map another schema (e.g. one of the ABCD extension ABCD-EFG, ABCD-DNA or HISPID)..." wikitext text/x-wiki This tutorial will explain how to set up a mapping for the ABCD2.06 data schema. If you intend to map another schema (e.g. one of the ABCD extension ABCD-EFG, ABCD-DNA or HISPID), you still should take the time to read it. Large portions of the process will be identical. == Access to Biological Collection Data Schema (ABCD) == ABCD in its current version 2.06 is the data schema typically used in conjunction with BioCASe. It is a highly complex XML data schema with about 1,000 elements that is able to store almost every piece of information that can be found in a natural history specimen collection or an observation database. The complexity is due to the fact that it can be used for a wide range of collections/databases – for living and preserved specimens, observations and culture collections, for zoological, botanical, bacterial and viral collections, marine or terrestrial, for herbaria, botanic and zoological gardens. For each of these special types, ABCD features special sections where information specific for this type can be stored. Other sections of ABCD will be shared by all these types, for example for gathering site, identifier or metadata. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a list of commonly used ABCD elements on the Wiki page [[CommonABCD2Concepts|Common ABCD2 Concepts]]. == Creating an ABCD Mapping for a Data Source == If you created a data source from a template (e.g. ''abcdmetadata''), a schema mapping for ABCD will be already existent. In this case, you can directly jump to the next section. If you created an empty data source, the section <small>Schemas</small> in the data source configuration overview will be empty: [[File:abcdSchemaMappings.png|border]] In order to create a new schema mapping, select the desired schema from the list and click <small>Create</small>. For ABCD, choose <small>ABCD_2.06.xml</small> (be careful not to hit one of the ABCD extensions, they look quite similar). After clicking <small>Create</small>, you will be directed to the mapping editor, which will allow you to edit the mappings. == The Mapping Editor == The mapping editor allows you to add, remove and edit ABCD mappings. On top you’ll find some summary information, on the bottom you’ll see the structure of the ABCD mapping: [[File:abcdMappingEditorEmpty.png|border]] First thing we will do is to set the root table, which is the table that holds the records you want to publish (specimens or observations). That table must hold one record per object to be published, identified by a unique identifier (called ''UnitID''). This identifier doesn't need to be the primary key necessarily, as long as it is unique (if that sentence confuses you, ignore it and just keep in mind that the UnitID must be unique.) Use the drop down box labelled <small>Root table Alias</small> to select the root table of your data model. If your metadata table holds just one record (because all occurrences published share the same metadata) and if it is not linked to the root table by a foreign key, you can specify this table in the drop-down box <small>Static table Alias</small>. The Provider Software will use a natural join to connect the metadata record stored in this table to all occurrences published. If your metadata table is referenced by a foreign key in the root able, you must not declare the metadata table as a static table! So either join the metadata table using a foreign key OR declare it as a static table. '''Note:''' If you're using Access, you cannot use a static table, because Access doesn't support that. So when using Access, you always have to reference the metadata table with a foreign key. When you’ve set the root table (and static table, if you use that), press <small>Save</small>. As with the DB structure setup, this will write your changes to the configuration files, closing the tab without saving will discard your changes! Also, in case you messed something up, you can use the <small>Revert</small> button to restore the state when you’ve saved for the last time. In the mapping editor, the link <small>Overview</small> at the top of the page will take you back to the datasource configuration overview. After creating a mapping, it will appear in the section <small>Schemas</small>. The entry will show the number of mapped elements (0 for now) and the schema namespace. Pressing the trash can symbol will move it to the trash can, which means that your web service does not support this schema anymore; if you did this by mistake, you can restore it by pressing the <small>Restore</small> button. <small>Purge</small> will delete the schema mapping permanently: [[File:abcdSchemaMappings2.png|border]] You can map several schemas for a given datasource. For example, in addition to the current ABCD2 standard, you could add the deprecated version 1.2 (even though I have no idea why you should do that) or DarwinCore. Just select the appropriate schema in the drop-down list and click <small>Create</small>. Since the namespace of a schema will be used in requests, you cannot create two mappings for the same namespace. Trying this will get you an error message. == Adding Mandatory ABCD Elements == Clicking on a schema in the datasource overview page will take you again to the mapping editor. In the lower half of the page, you will see the basic structure of an ABCD document: [[File:abcdMappingEditorEmptyBottom.png|border]] <strong>Technical/Content Contact:</strong> These trees hold information on who should be contacted for technical issues or questions concerning the published database, each with name, address, email address and telephone number. <strong>Metadata:</strong> In this sub tree you’ll find all metadata fields, for example title and description of the dataset, ownership information, date last modified, IPR statements etc. <strong>Units:</strong> This part stores the "real" data you want to publish, that means all specimen or observation data. The three elements already shown in the tree are identifiers that make up the so called "Triple ID", which uniquely identifies records within primary biodiversity networks such as BioCASe and GBIF: * ''UnitID'', which is a unique identifier for each record to be published (you can use the catalogue number, for example, or an existing barcode); * ''SourceInstitutionID'', an identifier for the institution publishing the dataset (for example BGBM); * ''SourceID'', an identifier for the dataset (e.g. “Herbarium Berolinense’’). The elements already shown in the mapping editor are printed in red because they are mandatory, which means they are required for an ABCD document. Without them being mapped (to database fields that are non-empty), the Provider Software will not be able to construct valid ABCD documents and will not publish any records. So the first elements that should be mapped are the mandatory fields. To add a mapping, click the <strong>+</strong> symbol next to the field, which will open the mapping editor dialog, where you can select the database table/column you want to map the ABCD element to: [[File:abcdMappingEditorDialog.png|border]] In this case, the element ''/DataSets/DataSet/Metadata/RevisionData/DateModified'' is mapped to the column ''source_update'' of the ''metadata'' table, which has the data type ''date''. (Remember that table/column retrieval is not supported for all DBMS, so if you’re unlucky, you might need to type in table/column name manually.) <small>Save</small> will accept your settings and close the dialog, <small>Cancel</small> will discard any changes. Instead of mapping an element to a database field, you can map it to a literal. For example, you could enter the technical and content contact information directly as literals in the mapping editor dialog (assuming they’re the same for all records). However, we strongly discourage you from doing this. For one thing, this is just bad style, since it will scatter the information published over different places: Some will be retrieved from the database, some (the literals) will be loaded from the configuration files. Changing the metadata of your service later would then require access to the database and access to the BioCASe configuration – that means the configuration tool password. Be nice to the person who will do these changes and put everything neatly in a database table! The second reason is even more important: Using literals for an element will result in this element to become not searchable, which means it cannot be used in a filter of a Search or a Scan request. Even though this behaviour is discoverable with a Capabilities request (by evaluating the ''searchable'' flag), a potential user of your web service might just not think of that, so it’s a good idea to avoid that in first place. Please use literals sparsely! In the mapping editor window, you can select several source columns and several literals, even a combination of both. The Provider Software will simply concatenate the values retrieved from the database and the literals specified (in the order they’re given). But be careful: any use of concatenation or literals will make the element not searchable. A better solution is to do such manipulations earlier, in a database view, and direct the mapping editor to a single resulting database field. Use the mapping editor window to add mappings for all eight mandatory fields: Click on the <strong>+</strong> symbol next to the element, choose table/column name and the correct data type, then press <small>Save</small>. When you’re done, the page should look similar to the following image. Press <small>Save</small> in the main window to write your changes to the configuration files. [[File:abcdMappingEditorMandatoryElements.png|border]] == Initial Testing of the Data Source == Once you’ve mapped all mandatory elements, you can do the first test of your BioCASe web service. Even though you haven’t mapped any real information yet, it is a good idea to check the service and solve any problems hitherto before continuing with the mapping. Click on <small>Test mapping!</small> to open the Query Form that can be used for this purpose: [[File:abcdQueryFormEmpty.png|border]] With the Query Form you can send a BioCASe request to your web service; it will wait for the reply of the web service and display the response document. On top of the page you can see the URL of your web service. In this case it is ''<nowiki>http://localhost/biocase/pywrapper.cgi?dsa=flora</nowiki>'', ''flora'' being the name of the datasource. Make sure this is correct for your installation! Into the text box below you can enter the request XML text. Because it would be cumbersome to enter the whole BioCASe request, you can use the links below the box to load templates. For now, click on <small>ABCD2 Search</small> to fill the box with a BioCASe Search request for documents in the ABCD2 format (the one you’ve just mapped): <syntaxhighlight> <?xml version='1.0' encoding='UTF-8'?> <request xmlns='http://www.biocase.org/schemas/protocol/1.3'> <header><type>search</type></header> <search> <requestFormat>http://www.tdwg.org/schemas/abcd/2.06</requestFormat> <responseFormat start='0' limit='10'>http://www.tdwg.org/schemas/abcd/2.06</responseFormat> <filter> <like path='/DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString'>A*</like> </filter> <count>false</count> </search> </request> </syntaxhighlight> As you can see, there is a filter set on the ''ScientificName'' element. Since we haven’t mapped this element yet, this won’t work. So let’s remove the filter (all lines between <filter> and </filter>, including the ''filter'' tags), leaving the text box like this: [[File:abcdQueryFormFilled.png|border]] Pressing <small>Submit</small> will send the request to the web service and – hopefully – display the response document with the correct ABCD records. For the mapping we’ve just created, it should look similar to the following (use the "–" symbol next to the trees TechnicalContact, ContentContact and Metadata to collapse them): [[File:abcdResponseMandatoryElements.png|border]] If you get an error, please check out the [[Debugging]] tutorial on how to debug a web service. You should continue the mapping process only after your web service returns results for the basal mapping we’ve just created. == Adding additional ABCD Elements == Once you’ve successfully managed to map the mandatory ABCD elements ("successfully" means the web service returns ABCD documents), you can go on with the mapping process. This is usually an iterative process: You add a handful of elements (for example, country, country code, locality text and coordinates for the gathering site), then you use the Query Form to see if the information ends up where it is supposed to do. This is the reason the Query Form opens up in a separate tab (or window, of you configured your web browser so): In one tab, you use the mapping editor to add new elements; after pressing <small>Save</small> you go to the Query Form tab and see what the response documents look like. Redo this until you’ve mapped all information you want to publish. In order to map new ABCD concepts, tick the check box labelled <small>Show all concepts</small> and press <small>Refresh</small>. The mapping editor will then display the whole ABCD tree, which looks admittedly a bit intimidating. In order to find the concept you’re looking for, use the search function of your browser. If you want to map the country name of the gathering site, for example, hit <small>Ctrl + F</small> (for Firefox, Microsoft IE and Opera) and type in the name of the ABCD element, namely ''country''. The browser will jump to the ''Country'' tree of the ''Gathering'' element, where you’ll find the elements for country name (''Name''), ISO country code (''ISO3166Code'') and the country name in the country’s language (''NameDerived''): [[File:abcdMappingEditorSearch.png|500px]] Continue the sequence mapping new elements/testing until you’ve mapped all information you want to publish. You can have a look at the list of recommend and commonly used ABCD elements on the [[CommonABCD2Concepts|Common ABCD2 Concepts]] page. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki]. The following page will show you a sample response ABCD document for a web service with 77 concepts mapped. Of course your documents will look different, because you will probably use different ABCD elements, but still it’s good to get an idea: [[SampleABCDDocument]]. fded962cffd9c05d77e91db6fe66c81b84caf732 0 23 570 569 2012-05-31T13:03:40Z JoergHoletschek 3 /* Version 3.2 [upcoming] */ wikitext text/x-wiki == Current Version 3 == === Version 3.2 [upcoming] === * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 2dc88e565b2d17c45b44cf29f399449cfb812399 571 570 2012-06-13T13:25:22Z JoergHoletschek 3 /* Version 3.2 [upcoming] */ wikitext text/x-wiki == Current Version 3 == === Version 3.2 [upcoming] === * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a mapping doesn't break the Capabilities any more (PSF is updated to reflect this change) === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool f3400e3a8c090fe693d1653ecaf0ef6396c3752c 572 571 2012-06-13T13:26:03Z JoergHoletschek 3 /* Version 3.2 [upcoming] */ wikitext text/x-wiki == Current Version 3 == === Version 3.2 [upcoming] === * DarwinCore Archives can now be generated directly from the config tool * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a mapping doesn't break the Capabilities any more (PSF is updated to reflect this change) === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 8313ab06a2331d3226e52ed47c3ce1fd36078f61 573 572 2012-06-13T13:32:42Z JoergHoletschek 3 /* Version 3.2 [upcoming] */ wikitext text/x-wiki == Current Version 3 == === Version 3.2 [upcoming] === * DarwinCore Archives can now be generated directly from the config tool * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool fe44ce26a08cf06bac84d743d8b599a104ea2753 0 6 574 474 2012-07-04T11:37:53Z JoergHoletschek 3 wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html === Testing the Link === Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a Postgres and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it: # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download and install the new version as described above (lucky enough, you can skip the installation of Apache, Python and any DB packages) into a directory with the name you just memorised. When running the setup script, make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). Try out your new installation to make sure it is working. Before pointing your web server to the new installation, manually copy the configuration files located in the config folder of your old installation to the config folder of your new installation: config.ini for the system settings and all folders in the datasources directory. You can then edit new installation and test the web services. 94f886528d90a8bac5a2f6d564717f938e5e387a 575 574 2012-07-04T11:49:14Z JoergHoletschek 3 /* Updating an existing BioCASe Installation */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html === Testing the Link === Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a Postgres and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it: # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # Run the setup script. Provide the the same base URL that was used by the old installation (for example ''/biocase''). # Change the default password of the new installation to the one you used in the old installation. # Open the folder ''config'' of your old installation and open the file ''config.ini'' in a text editor. Use the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values you don't see in the old file unchanged). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to the folder ''config/datasources'' of your new installation. Done! Try out your new installation to make sure it is working. af3a2f594623b846bd01b0a9fe8f9df6eb7bff95 576 575 2012-07-04T11:51:29Z JoergHoletschek 3 /* Updating an existing BioCASe Installation */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html === Testing the Link === Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a Postgres and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it: # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Run the setup script]]. Provide the the same base URL that was used by the old installation (for example ''/biocase''). # Change the default password of the new installation to the one you used in the old installation. # Open the folder ''config'' of your old installation and open the file ''config.ini'' in a text editor. Use the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values you don't see in the old file unchanged). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to the folder ''config/datasources'' of your new installation. Done! Try out your new installation to make sure it is working. ea036942677d1a04ae1455c311458ef11619bd03 577 576 2012-07-04T11:52:12Z JoergHoletschek 3 /* Updating an existing BioCASe Installation */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html === Testing the Link === Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a Postgres and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it: # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Running_the_Setup_Script|Run the setup script]]. Provide the the same base URL that was used by the old installation (for example ''/biocase''). # Change the default password of the new installation to the one you used in the old installation. # Open the folder ''config'' of your old installation and open the file ''config.ini'' in a text editor. Use the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values you don't see in the old file unchanged). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to the folder ''config/datasources'' of your new installation. Done! Try out your new installation to make sure it is working. 69e1d3db7ad8ea56eb9aef3d17494f1153fde212 578 577 2012-07-04T11:52:44Z JoergHoletschek 3 /* Updating an existing BioCASe Installation */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html === Testing the Link === Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a Postgres and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it: # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. Provide the the same base URL that was used by the old installation (for example ''/biocase''). # Change the default password of the new installation to the one you used in the old installation. # Open the folder ''config'' of your old installation and open the file ''config.ini'' in a text editor. Use the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values you don't see in the old file unchanged). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to the folder ''config/datasources'' of your new installation. Done! Try out your new installation to make sure it is working. 391a8248bf80879ad151dfd269da8c12d8cd79b2 579 578 2012-07-04T11:53:23Z JoergHoletschek 3 /* Updating an existing BioCASe Installation */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html === Testing the Link === Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a Postgres and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it: # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. Provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the folder ''config'' of your old installation and open the file ''config.ini'' in a text editor. Use the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values you don't see in the old file unchanged). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to the folder ''config/datasources'' of your new installation. Done! Try out your new installation to make sure it is working. a9ed0abf36ee5ad6060337fc2c12e450392fe62a 580 579 2012-07-04T11:54:24Z JoergHoletschek 3 /* Updating an existing BioCASe Installation */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html === Testing the Link === Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a Postgres and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it: # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. Provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values you don't see in the old file unchanged). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to the folder ''config/datasources'' of your new installation. Done! Try out your new installation to make sure it is working. 0df870ff87206b31092af6079ed926e97cc14873 581 580 2012-07-04T11:54:50Z JoergHoletschek 3 /* Updating an existing BioCASe Installation */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html === Testing the Link === Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a Postgres and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it: # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. Provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values you don't see in the old file unchanged). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. Done! Try out your new installation to make sure it is working. d6cedfe8ef01b6db8cd97883b8f99a663a47b2cc 582 581 2012-07-04T11:56:16Z JoergHoletschek 3 /* Updating an existing BioCASe Installation */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html === Testing the Link === Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a Postgres and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == First: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it: # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. Provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values you don't see in the old file unchanged). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. 2aa0e8ecc84d24ccb23975c3f09fdb596dc7b04f 585 582 2012-07-04T12:03:33Z JoergHoletschek 3 /* Updating an existing BioCASe Installation */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html === Testing the Link === Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a Postgres and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == Important: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. Provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values you don't see in the old file unchanged). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. df0eb547c383dcf99bcd7cded114a03b3ec61022 586 585 2012-07-04T12:05:00Z JoergHoletschek 3 /* Updating an existing BioCASe Installation */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html === Testing the Link === Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a Postgres and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == Important: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. Provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. 1772ffdfd7ff46228f1931832a0a7259faee9315 587 586 2012-07-05T09:16:07Z JoergHoletschek 3 /* Installing Additional Python Packages */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html === Testing the Link === Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages/Java == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for both the Logs and the Configuration. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a Postgres and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == Important: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. Provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. bdde8b3d2e06b55e87840eae5dcde58d42548abf 589 587 2012-07-05T09:26:11Z JoergHoletschek 3 /* Checking Your Installation */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html === Testing the Link === Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages/Java == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all 4 drectories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a Postgres and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == Important: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. Provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. 8d35fc967aa81c2576e8d11f909d55c45b38b916 590 589 2012-07-05T09:26:32Z JoergHoletschek 3 /* Checking Your Installation */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html === Testing the Link === Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages/Java == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a Postgres and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == Important: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. Provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. c069d6b8fdf069366a89ffd1480baeb1a70e8317 591 590 2012-07-05T09:28:33Z JoergHoletschek 3 /* Checking Your Installation */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html === Testing the Link === Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages/Java == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == Important: If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. Provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ab3c6111b564eba60da22c342665246507596e99 0 52 583 476 2012-07-04T11:57:43Z JoergHoletschek 3 /* I want to update my BioCASe installation. How do I do this? */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I'm using Internet Information Server, and after creating the Virtual Directory for BioCASe, I get a “502.2 - Bad Gateway. The specified CGI application misbehaved by not returning a complete set of HTTP headers” error.=== This happens when you didn't correctly associate .cgi and .py files with Python. Make sure you've linked both files with Python and that the path contains the placeholders ''%s %s'' (see [[Installation#Internet_Information_Server|Linking IIS]]). ===I cannot find the installation package of the database connectivity module for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later of psycopg2. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== See [[Installation#Updating_an_existing_BioCASe_Installation|Updating_an_existing_BioCASe_Installation]]. ===I've updated my installation from a version prior to 2.6. Now, when I enter the DB structure, some of the table and column names are gone, the same happens in the mapping editor.=== Starting with version 2.6, BioCASe retrieves table and column names from the database for some database management systems (namely MySQL, SQL Server, Postgres, Foxpro, Access and Oracle). In older versions, you had to enter table/column names manually, and when you did the setup with such an early version, you probably didn't care for upper/lower cases. Since some DB management systems are case-sensitive, BioCASe can't ignore the differences and won't find these technically incorrect names in the lists retrieved from the database. The respective drop down boxes will therefore be empty. If that happens, just select the correct names from the drop down lists. ===When I try to create an archive, I get a "Permission denied" error. Why is that?=== For creating archives, you need to grant write permissions for the folders ''archive'' and ''www/downloads'' of your BioCASe installation to Python. You can check if permissions are set correctly in the test libs page (Start --> Utilities --> Test Libs). == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other only by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===I've mapped ABCD2. When testing the web service, I get a negative number for "RecordsDropped" in the response document. What does that mean?=== Same problem as above - it means you have an error in your mapping. Most probably you've mapped a non-repeatable ABCD concept to a table/column in your database that holds several entries per unit (occurrence) record. The BioCASe Provider Software will duplicate the respective unit records to be able to create valid ABCD documents - which is indicated by a negative number of "records dropped". Make sure the cardinality of your database relations match the structure (in respects of element repeatability) of ABCD to avoid this problem. ===I'm using pymssql to connect to a SQL Server database. Char(n) and Varchar(n) strings get truncated to 255 characters - what's wrong?=== This is known limitation of TDS protocol used by pymssql. Just change the data type for columns storing long string to TEXT or create a view that converts Char/Varchar columns to TEXT columns on the fly to work around this issue. ===I'm using pymssql to connect to a SQL Server database, BioCASe is running on a Linux distribution. When I do a Search or Scan request, I don't get any results (0 hits), even though there are definitely records in the database.=== That problem occurs on old versions of pymssql (1.0.2 or earlier) on Linux distributions. Update to version 2.0 or later to solve this. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. == Trouble Shooting == ===I’ve forgotten the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. fc7f15efd3da0c7f6a6bbcd6b8a9f7e299061815 584 583 2012-07-04T11:58:04Z JoergHoletschek 3 /* I want to update my BioCASe installation. How do I do this? */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I'm using Internet Information Server, and after creating the Virtual Directory for BioCASe, I get a “502.2 - Bad Gateway. The specified CGI application misbehaved by not returning a complete set of HTTP headers” error.=== This happens when you didn't correctly associate .cgi and .py files with Python. Make sure you've linked both files with Python and that the path contains the placeholders ''%s %s'' (see [[Installation#Internet_Information_Server|Linking IIS]]). ===I cannot find the installation package of the database connectivity module for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later of psycopg2. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== See [[Installation#Updating_an_existing_BioCASe_Installation|Updating an existing BioCASe Installation]]. ===I've updated my installation from a version prior to 2.6. Now, when I enter the DB structure, some of the table and column names are gone, the same happens in the mapping editor.=== Starting with version 2.6, BioCASe retrieves table and column names from the database for some database management systems (namely MySQL, SQL Server, Postgres, Foxpro, Access and Oracle). In older versions, you had to enter table/column names manually, and when you did the setup with such an early version, you probably didn't care for upper/lower cases. Since some DB management systems are case-sensitive, BioCASe can't ignore the differences and won't find these technically incorrect names in the lists retrieved from the database. The respective drop down boxes will therefore be empty. If that happens, just select the correct names from the drop down lists. ===When I try to create an archive, I get a "Permission denied" error. Why is that?=== For creating archives, you need to grant write permissions for the folders ''archive'' and ''www/downloads'' of your BioCASe installation to Python. You can check if permissions are set correctly in the test libs page (Start --> Utilities --> Test Libs). == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other only by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===I've mapped ABCD2. When testing the web service, I get a negative number for "RecordsDropped" in the response document. What does that mean?=== Same problem as above - it means you have an error in your mapping. Most probably you've mapped a non-repeatable ABCD concept to a table/column in your database that holds several entries per unit (occurrence) record. The BioCASe Provider Software will duplicate the respective unit records to be able to create valid ABCD documents - which is indicated by a negative number of "records dropped". Make sure the cardinality of your database relations match the structure (in respects of element repeatability) of ABCD to avoid this problem. ===I'm using pymssql to connect to a SQL Server database. Char(n) and Varchar(n) strings get truncated to 255 characters - what's wrong?=== This is known limitation of TDS protocol used by pymssql. Just change the data type for columns storing long string to TEXT or create a view that converts Char/Varchar columns to TEXT columns on the fly to work around this issue. ===I'm using pymssql to connect to a SQL Server database, BioCASe is running on a Linux distribution. When I do a Search or Scan request, I don't get any results (0 hits), even though there are definitely records in the database.=== That problem occurs on old versions of pymssql (1.0.2 or earlier) on Linux distributions. Update to version 2.0 or later to solve this. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. == Trouble Shooting == ===I’ve forgotten the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. 8a7254fdedd379b6a1be43298fcff2e503908eb4 6 14 588 30 2012-07-05T09:23:42Z JoergHoletschek 3 uploaded a new version of &quot;[[File:InstLibrariesVanilla.png]]&quot; wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 16 592 34 2012-07-05T09:36:41Z JoergHoletschek 3 uploaded a new version of &quot;[[File:InstLibrariesDone.png]]&quot; wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 0 49 593 303 2012-07-05T09:38:50Z JoergHoletschek 3 /* Requirements for Using the BPS */ wikitext text/x-wiki == What is the BioCASe Provider Software? == The BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. The Provider Software is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. The Provider Software is a suite of tools that need to be installed on the data provider’s web server. The core component is the PyWrapper, an XML/CGI database interface written in Python that allows a standardized access to a variety of database management systems and arbitrarily structured databases. Grouped around this are a number of tools for configuring, testing and debugging a BioCASe installation or web service. Even though BioCASe can be used for any conceptual XML data schema, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the [http://www.biocase.org BioCASe network] and the [http://www.gbif.org Global Biodiversity Information Facility]. == Requirements for Using the BPS == Before using the BioCASe Provider Software, three requirements need to be checked: * For one thing, the data to be published needs to be stored in an SQL compliant relational database management system (DBMS), with the data model being well documented or at least knowledge about the data model existing on someone’s head. Non-SQL DBMS are not supported. Currently, the BioCASe Provider Software can connect to the following databases: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * For installing BioCASe, you need to have a web server (Apache or Microsoft IIS) that is permanently connected to the Internet. This can be an institutional server or a hosting service (even though it’s usually a pain to get privileges to install software for the latter). Dial-up connections won’t work. * For the web server in question, you must have the possibility to install Python (if it is not installed yet) and Python packages. Even though this can be done by the IT staff, you should make sure that there are no institutional policies colliding. If you want to create DarwinCore Archives, a Java Runtime Environment is required. == Steps == Once you’ve made sure the BioCASe Provider Software is the right choice for you ([[Special:Contact|contact the BioCASe team]] if you feel unsafe with your decision), you need to do the following steps: # Think about which information you want to publish to the network. This involves settling the question of ownership of the data and the terms of use, license and usage restrictions for the published data. Also you should consider which information should '''not''' be published because they might affect endangered species (options are to block certain occurrences from publication or to blur the exact locality information). # Prepare the database for publication as described in the [[Preparation]] tutorial. This includes the decision on whether to publish the live database or a snapshot, the creation of a table for metadata and, in certain cases, some data transformation. # Follow the [[Installation]] Tutorial to install the BioCASe Provider Software. # Set up a BioCASe Datasource and connect it to your database, then configure the database structure as described in the [[DatasourceSetup]] tutorial. # Follow the [[ABCD2Mapping]] tutorial to create a mapping for the schema you want to use. # Test and debug your BioCASe web service. Read the [[Debugging]] tutorial to learn how to do this. # Once you’re done with setting up and configuring the BioCASe Provider Software, you can register the BioCASe web service with the information network you want to publish your data to, for example with [http://www.gbif.org GBIF]. We strongly recommend to [[Special:Contact|contact the BioCASe team]] for checking your web service first as we might be able to foresee problems that are more difficult to resolve after registration. == Support == In case you run into difficulties during the installation and setup, you should * have a look at the [[FAQ]] which might list the solution for your question, * ask other BioCASe data providers for help, * get in contact with the [[Special:Contact|BioCASe team]]. Please describe exactly you environment (operating system, DBMS used), the steps that cause the problem and any error messages and hints you get. Screenshot are a big plus here. 5c2c0763327c68e7eccd879e65181258ec09897f 0 80 594 439 2012-07-05T09:52:13Z JoergHoletschek 3 wikitext text/x-wiki With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:archArchivingPage.png|border]] ===Usage=== In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create a new Archive</small> allows you to create a new archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:archProcess.png|border]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== XML Archives that use the ABCD 2.06 schema can be converted into DarwinCore Archives. Currently, this feature is only available as a standalone command-line tool. Once thoroughly tested, it will be intregrated into the BioCASe Provider Software and can be trigged from the configuration tool. ===Requirements=== Since this tool is currently a standalone application, it doesn't need to be run on the same machine as the BioCASe Provider Software. But of course it can - if you're running BioCASe on a publicly available server and plan to use DwCAs for production, the BioCASe server is definitely the right place. * Java Runtime Environment 1.5 or later installed * 2+G of RAM available * 100M free disk space * One or several ABCD2.06 Archives (created with BPS3+) ===Installation=== # If you don't have a Java Runtime Environment (version 1.5 or later) installed on your machine, get one [http://www.java.com/de/download/ here]. # Download the DwCA Creator from the [http://www.biocase.org/products/provider_software/index.shtml#DwCA_Creator BioCASe Website] and unzip the archive file to your machine. This will create a folder ''dwca'' for the tool. # Test the installation by opening a terminal (''command prompt'' on Windows) and changing to the ''dwca'' folder. Start the tool by running the batch file ''dwca.bat'' on Windows or the shell script ''dwca.sh'' on Linux/MacOS. Since it doesn't do any processing, the test run should take only a few seconds and print some lines similar to following (you can ignore the warning ''No such file or directory'' in the last line): [[File:dwcaTest.png]] ===Usage=== The converter can be started with the scripts ''dwca.bat'' (on Windows) and ''dwca.sh'' (on Linux/MacOS) and usually takes one argument - the ABCD archive to be converted. The tool accepts both a file stored in the local file system and an archive available for download at a given URL. Let's assume you've created an ABCD2 dump for your collection. You've placed the archive file (named ''AlgaTerra_ABCD_2.06.zip'') in the same folder as the tool (''dwca''). To run the conversion on Linux, open a terminal window, change to the ''dwca'' folder and type in the following: <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command. Note that the base folder for the transformer is the ''kettle'' folder, so you need to specify ''..\'' to direct the transformer to its parent folder for finding the archive file (alternatively, you can put the archive file in the ''kettle'' subfolder and thus spare the ''..\''): <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip</pre> This will unzip the ABCD documents stored in the archive file, create the data files, EML document(s) and the archive descriptor(s): [[File:dwcaStartLocal.png]] Depending on the size of your collection (number of records), the amount of information published (number of concepts mapped) and the speed of your server (available memory, number of CPUs/cores), this will take from several seconds to some minutes. For large collections (millions of records), it can take up to hours. After each chunk of 50,000 records processed, a line indicating the progress will be printed, so you will be able to see if the tool is still running. However, independent of the amount of data to be processed, the last lines of the output will look similar to the following: [[File:dwcaEndLocal.png]] When creating an ABCD Archive with the BioCASe Provider Software, it becomes automatically available for download on the BioCASe installation after the process has finished. The DwCA Creator can download the archive directly from this location. So if you're BioCASe installation is accessible from the machine you're running the prototype on, you can spare the trouble of moving the file manually and provide the URL instead of a local file name. To get the download URL for an ABCD Archive you've created, find the archive in question in the list of existing archives in the configuration tool (see [[Archiving#XML_Archives|here]] for more on that). Right-click on the Download link and choose ''Copy link''. Then open a terminal window for the creator tool and provide the link intead of a file name: <pre>./dwca.sh http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command: <pre>dwca.bat http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> ===Output=== The DwCA(s) created can be found in the folder ''output'' created in the base directory (''dwca''). Remember that there will be one DwCA per dataset. So depending upon how many datasets are stored in the source ABCD archive, there will be one or several files in the output folder. On Linux and MacOS, the shell script ''dwca.sh'' will zip all files that make up one DarwinCore Archive into an archive file named after the dataset title. So you could send this files directly to a potential DwCA consumer: [[File:dwcaOutputFolderLinux.png]] On Windows, the batch file ''dwca.bat'' of the prototype doesn't do that zipping. So if you go to the output file after the conversion has finished, you will see several text files and XML documents, all named after the pattern ''<dataset> <filename>''. So for the example above, the output folder would look like this: [[File:dwcaOutputFolderWindows.png]] In order to create a DwCA that can be sent to a consumer, please do the follwing for each ''<dataset>'': # Note down the dataset title, # Remove the dataset title in the file names (''<dataset> <filename>'' becomes ''<filename>''), # Create a zip named ''<dataset>.zip'' with all the files you just renamed. This lack of functionality is due to the limited capabilities of Windows batch files. Once integrated into the Provider Software, you won't need to zip up files manually. ===Trouble Shooting=== If you run into problems when using this tool, please [[Special:Contact|contact us]] for help. The default settings of the transormation should be working for small and medium-sized collections (up to 500,000 records). If the processing is cancelled with an OutOfMemoryError, you should try the follwing: '''Increasing the maximum memory used by the transformer'''<br> Per default, the transformer will use up to 1GB of RAM for the transformation process. In case of an OutOfMemoryError, you should increase this limit to 2GB. Don't be stingy with memory, since it will be only used during the tranformation process and be freed afterwards. Moreover, this is only the upper limit, so actual memory usage might well stay below this limit. To allow the transformer to use more memory, go to the folder ''kettle'' of your installation. Open the file ''kitchen.sh'' (Linux/MacOS) or ''kitchen.bat'' (Windows) in a text editor. Use the ''Find'' feature of the editor to go to the spot in file that specifies the memory limit. Replace the value ''1024'' with the amount of memory you want to allow the transformation to use, so e.g. ''2048'' for 2GB: [[File:dwcaSettingMemory.png]] Save the file and restart the transformation. '''Lowering the number of records stored in memory'''<br> Per default, Kettle will store up to 100,000 records in memory before starting to use temporary files for sorting rows. You can set this threshold to a lower value to save some memory. However, be advised that this will result in heavy disk I/O on your machine, slowing down the transformation. Therefore you should first increase the memory limit before changing this setting. If you decide to do this, just add the desired maximim number of records to be stored in memory as a second parameter when invoking the transformer. So on Linux, type in <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip 10000</pre> in order to set the threshold to 10,000 records. On a Windows machine, type in following: <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip 10000</pre> 73b512664de9db1b458c62df4bde1e5303ef7b36 596 594 2012-07-05T10:50:46Z JoergHoletschek 3 wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:archArchivingPage.png|border]] ===Usage=== In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create a new Archive</small> allows you to create a new archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:archProcess.png|border]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== XML Archives that use the ABCD 2.06 schema can be converted into DarwinCore Archives. Currently, this feature is only available as a standalone command-line tool. Once thoroughly tested, it will be intregrated into the BioCASe Provider Software and can be trigged from the configuration tool. ===Requirements=== Since this tool is currently a standalone application, it doesn't need to be run on the same machine as the BioCASe Provider Software. But of course it can - if you're running BioCASe on a publicly available server and plan to use DwCAs for production, the BioCASe server is definitely the right place. * Java Runtime Environment 1.5 or later installed * 2+G of RAM available * 100M free disk space * One or several ABCD2.06 Archives (created with BPS3+) ===Installation=== # If you don't have a Java Runtime Environment (version 1.5 or later) installed on your machine, get one [http://www.java.com/de/download/ here]. # Download the DwCA Creator from the [http://www.biocase.org/products/provider_software/index.shtml#DwCA_Creator BioCASe Website] and unzip the archive file to your machine. This will create a folder ''dwca'' for the tool. # Test the installation by opening a terminal (''command prompt'' on Windows) and changing to the ''dwca'' folder. Start the tool by running the batch file ''dwca.bat'' on Windows or the shell script ''dwca.sh'' on Linux/MacOS. Since it doesn't do any processing, the test run should take only a few seconds and print some lines similar to following (you can ignore the warning ''No such file or directory'' in the last line): [[File:dwcaTest.png]] ===Usage=== The converter can be started with the scripts ''dwca.bat'' (on Windows) and ''dwca.sh'' (on Linux/MacOS) and usually takes one argument - the ABCD archive to be converted. The tool accepts both a file stored in the local file system and an archive available for download at a given URL. Let's assume you've created an ABCD2 dump for your collection. You've placed the archive file (named ''AlgaTerra_ABCD_2.06.zip'') in the same folder as the tool (''dwca''). To run the conversion on Linux, open a terminal window, change to the ''dwca'' folder and type in the following: <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command. Note that the base folder for the transformer is the ''kettle'' folder, so you need to specify ''..\'' to direct the transformer to its parent folder for finding the archive file (alternatively, you can put the archive file in the ''kettle'' subfolder and thus spare the ''..\''): <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip</pre> This will unzip the ABCD documents stored in the archive file, create the data files, EML document(s) and the archive descriptor(s): [[File:dwcaStartLocal.png]] Depending on the size of your collection (number of records), the amount of information published (number of concepts mapped) and the speed of your server (available memory, number of CPUs/cores), this will take from several seconds to some minutes. For large collections (millions of records), it can take up to hours. After each chunk of 50,000 records processed, a line indicating the progress will be printed, so you will be able to see if the tool is still running. However, independent of the amount of data to be processed, the last lines of the output will look similar to the following: [[File:dwcaEndLocal.png]] When creating an ABCD Archive with the BioCASe Provider Software, it becomes automatically available for download on the BioCASe installation after the process has finished. The DwCA Creator can download the archive directly from this location. So if you're BioCASe installation is accessible from the machine you're running the prototype on, you can spare the trouble of moving the file manually and provide the URL instead of a local file name. To get the download URL for an ABCD Archive you've created, find the archive in question in the list of existing archives in the configuration tool (see [[Archiving#XML_Archives|here]] for more on that). Right-click on the Download link and choose ''Copy link''. Then open a terminal window for the creator tool and provide the link intead of a file name: <pre>./dwca.sh http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command: <pre>dwca.bat http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> ===Output=== The DwCA(s) created can be found in the folder ''output'' created in the base directory (''dwca''). Remember that there will be one DwCA per dataset. So depending upon how many datasets are stored in the source ABCD archive, there will be one or several files in the output folder. On Linux and MacOS, the shell script ''dwca.sh'' will zip all files that make up one DarwinCore Archive into an archive file named after the dataset title. So you could send this files directly to a potential DwCA consumer: [[File:dwcaOutputFolderLinux.png]] On Windows, the batch file ''dwca.bat'' of the prototype doesn't do that zipping. So if you go to the output file after the conversion has finished, you will see several text files and XML documents, all named after the pattern ''<dataset> <filename>''. So for the example above, the output folder would look like this: [[File:dwcaOutputFolderWindows.png]] In order to create a DwCA that can be sent to a consumer, please do the follwing for each ''<dataset>'': # Note down the dataset title, # Remove the dataset title in the file names (''<dataset> <filename>'' becomes ''<filename>''), # Create a zip named ''<dataset>.zip'' with all the files you just renamed. This lack of functionality is due to the limited capabilities of Windows batch files. Once integrated into the Provider Software, you won't need to zip up files manually. ===Trouble Shooting=== If you run into problems when using this tool, please [[Special:Contact|contact us]] for help. The default settings of the transormation should be working for small and medium-sized collections (up to 500,000 records). If the processing is cancelled with an OutOfMemoryError, you should try the follwing: '''Increasing the maximum memory used by the transformer'''<br> Per default, the transformer will use up to 1GB of RAM for the transformation process. In case of an OutOfMemoryError, you should increase this limit to 2GB. Don't be stingy with memory, since it will be only used during the tranformation process and be freed afterwards. Moreover, this is only the upper limit, so actual memory usage might well stay below this limit. To allow the transformer to use more memory, go to the folder ''kettle'' of your installation. Open the file ''kitchen.sh'' (Linux/MacOS) or ''kitchen.bat'' (Windows) in a text editor. Use the ''Find'' feature of the editor to go to the spot in file that specifies the memory limit. Replace the value ''1024'' with the amount of memory you want to allow the transformation to use, so e.g. ''2048'' for 2GB: [[File:dwcaSettingMemory.png]] Save the file and restart the transformation. '''Lowering the number of records stored in memory'''<br> Per default, Kettle will store up to 100,000 records in memory before starting to use temporary files for sorting rows. You can set this threshold to a lower value to save some memory. However, be advised that this will result in heavy disk I/O on your machine, slowing down the transformation. Therefore you should first increase the memory limit before changing this setting. If you decide to do this, just add the desired maximim number of records to be stored in memory as a second parameter when invoking the transformer. So on Linux, type in <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip 10000</pre> in order to set the threshold to 10,000 records. On a Windows machine, type in following: <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip 10000</pre> 05634e7c5b5d9e6c22cceae9b86869281fd3f218 597 596 2012-07-05T10:52:09Z JoergHoletschek 3 /* XML Archives */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:xmlArchives.png|border]] ===Usage=== In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create a new Archive</small> allows you to create a new archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:archProcess.png|border]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== XML Archives that use the ABCD 2.06 schema can be converted into DarwinCore Archives. Currently, this feature is only available as a standalone command-line tool. Once thoroughly tested, it will be intregrated into the BioCASe Provider Software and can be trigged from the configuration tool. ===Requirements=== Since this tool is currently a standalone application, it doesn't need to be run on the same machine as the BioCASe Provider Software. But of course it can - if you're running BioCASe on a publicly available server and plan to use DwCAs for production, the BioCASe server is definitely the right place. * Java Runtime Environment 1.5 or later installed * 2+G of RAM available * 100M free disk space * One or several ABCD2.06 Archives (created with BPS3+) ===Installation=== # If you don't have a Java Runtime Environment (version 1.5 or later) installed on your machine, get one [http://www.java.com/de/download/ here]. # Download the DwCA Creator from the [http://www.biocase.org/products/provider_software/index.shtml#DwCA_Creator BioCASe Website] and unzip the archive file to your machine. This will create a folder ''dwca'' for the tool. # Test the installation by opening a terminal (''command prompt'' on Windows) and changing to the ''dwca'' folder. Start the tool by running the batch file ''dwca.bat'' on Windows or the shell script ''dwca.sh'' on Linux/MacOS. Since it doesn't do any processing, the test run should take only a few seconds and print some lines similar to following (you can ignore the warning ''No such file or directory'' in the last line): [[File:dwcaTest.png]] ===Usage=== The converter can be started with the scripts ''dwca.bat'' (on Windows) and ''dwca.sh'' (on Linux/MacOS) and usually takes one argument - the ABCD archive to be converted. The tool accepts both a file stored in the local file system and an archive available for download at a given URL. Let's assume you've created an ABCD2 dump for your collection. You've placed the archive file (named ''AlgaTerra_ABCD_2.06.zip'') in the same folder as the tool (''dwca''). To run the conversion on Linux, open a terminal window, change to the ''dwca'' folder and type in the following: <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command. Note that the base folder for the transformer is the ''kettle'' folder, so you need to specify ''..\'' to direct the transformer to its parent folder for finding the archive file (alternatively, you can put the archive file in the ''kettle'' subfolder and thus spare the ''..\''): <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip</pre> This will unzip the ABCD documents stored in the archive file, create the data files, EML document(s) and the archive descriptor(s): [[File:dwcaStartLocal.png]] Depending on the size of your collection (number of records), the amount of information published (number of concepts mapped) and the speed of your server (available memory, number of CPUs/cores), this will take from several seconds to some minutes. For large collections (millions of records), it can take up to hours. After each chunk of 50,000 records processed, a line indicating the progress will be printed, so you will be able to see if the tool is still running. However, independent of the amount of data to be processed, the last lines of the output will look similar to the following: [[File:dwcaEndLocal.png]] When creating an ABCD Archive with the BioCASe Provider Software, it becomes automatically available for download on the BioCASe installation after the process has finished. The DwCA Creator can download the archive directly from this location. So if you're BioCASe installation is accessible from the machine you're running the prototype on, you can spare the trouble of moving the file manually and provide the URL instead of a local file name. To get the download URL for an ABCD Archive you've created, find the archive in question in the list of existing archives in the configuration tool (see [[Archiving#XML_Archives|here]] for more on that). Right-click on the Download link and choose ''Copy link''. Then open a terminal window for the creator tool and provide the link intead of a file name: <pre>./dwca.sh http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command: <pre>dwca.bat http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> ===Output=== The DwCA(s) created can be found in the folder ''output'' created in the base directory (''dwca''). Remember that there will be one DwCA per dataset. So depending upon how many datasets are stored in the source ABCD archive, there will be one or several files in the output folder. On Linux and MacOS, the shell script ''dwca.sh'' will zip all files that make up one DarwinCore Archive into an archive file named after the dataset title. So you could send this files directly to a potential DwCA consumer: [[File:dwcaOutputFolderLinux.png]] On Windows, the batch file ''dwca.bat'' of the prototype doesn't do that zipping. So if you go to the output file after the conversion has finished, you will see several text files and XML documents, all named after the pattern ''<dataset> <filename>''. So for the example above, the output folder would look like this: [[File:dwcaOutputFolderWindows.png]] In order to create a DwCA that can be sent to a consumer, please do the follwing for each ''<dataset>'': # Note down the dataset title, # Remove the dataset title in the file names (''<dataset> <filename>'' becomes ''<filename>''), # Create a zip named ''<dataset>.zip'' with all the files you just renamed. This lack of functionality is due to the limited capabilities of Windows batch files. Once integrated into the Provider Software, you won't need to zip up files manually. ===Trouble Shooting=== If you run into problems when using this tool, please [[Special:Contact|contact us]] for help. The default settings of the transormation should be working for small and medium-sized collections (up to 500,000 records). If the processing is cancelled with an OutOfMemoryError, you should try the follwing: '''Increasing the maximum memory used by the transformer'''<br> Per default, the transformer will use up to 1GB of RAM for the transformation process. In case of an OutOfMemoryError, you should increase this limit to 2GB. Don't be stingy with memory, since it will be only used during the tranformation process and be freed afterwards. Moreover, this is only the upper limit, so actual memory usage might well stay below this limit. To allow the transformer to use more memory, go to the folder ''kettle'' of your installation. Open the file ''kitchen.sh'' (Linux/MacOS) or ''kitchen.bat'' (Windows) in a text editor. Use the ''Find'' feature of the editor to go to the spot in file that specifies the memory limit. Replace the value ''1024'' with the amount of memory you want to allow the transformation to use, so e.g. ''2048'' for 2GB: [[File:dwcaSettingMemory.png]] Save the file and restart the transformation. '''Lowering the number of records stored in memory'''<br> Per default, Kettle will store up to 100,000 records in memory before starting to use temporary files for sorting rows. You can set this threshold to a lower value to save some memory. However, be advised that this will result in heavy disk I/O on your machine, slowing down the transformation. Therefore you should first increase the memory limit before changing this setting. If you decide to do this, just add the desired maximim number of records to be stored in memory as a second parameter when invoking the transformer. So on Linux, type in <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip 10000</pre> in order to set the threshold to 10,000 records. On a Windows machine, type in following: <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip 10000</pre> 41bc72f7310ebc5280794ca7a49efe6eecf04bbe 600 597 2012-07-05T10:58:30Z JoergHoletschek 3 /* XML Archives */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:xmlArchives.png|border|750px]] ===Usage=== In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create a new Archive</small> allows you to create a new archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:archProcess.png|border]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== XML Archives that use the ABCD 2.06 schema can be converted into DarwinCore Archives. Currently, this feature is only available as a standalone command-line tool. Once thoroughly tested, it will be intregrated into the BioCASe Provider Software and can be trigged from the configuration tool. ===Requirements=== Since this tool is currently a standalone application, it doesn't need to be run on the same machine as the BioCASe Provider Software. But of course it can - if you're running BioCASe on a publicly available server and plan to use DwCAs for production, the BioCASe server is definitely the right place. * Java Runtime Environment 1.5 or later installed * 2+G of RAM available * 100M free disk space * One or several ABCD2.06 Archives (created with BPS3+) ===Installation=== # If you don't have a Java Runtime Environment (version 1.5 or later) installed on your machine, get one [http://www.java.com/de/download/ here]. # Download the DwCA Creator from the [http://www.biocase.org/products/provider_software/index.shtml#DwCA_Creator BioCASe Website] and unzip the archive file to your machine. This will create a folder ''dwca'' for the tool. # Test the installation by opening a terminal (''command prompt'' on Windows) and changing to the ''dwca'' folder. Start the tool by running the batch file ''dwca.bat'' on Windows or the shell script ''dwca.sh'' on Linux/MacOS. Since it doesn't do any processing, the test run should take only a few seconds and print some lines similar to following (you can ignore the warning ''No such file or directory'' in the last line): [[File:dwcaTest.png]] ===Usage=== The converter can be started with the scripts ''dwca.bat'' (on Windows) and ''dwca.sh'' (on Linux/MacOS) and usually takes one argument - the ABCD archive to be converted. The tool accepts both a file stored in the local file system and an archive available for download at a given URL. Let's assume you've created an ABCD2 dump for your collection. You've placed the archive file (named ''AlgaTerra_ABCD_2.06.zip'') in the same folder as the tool (''dwca''). To run the conversion on Linux, open a terminal window, change to the ''dwca'' folder and type in the following: <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command. Note that the base folder for the transformer is the ''kettle'' folder, so you need to specify ''..\'' to direct the transformer to its parent folder for finding the archive file (alternatively, you can put the archive file in the ''kettle'' subfolder and thus spare the ''..\''): <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip</pre> This will unzip the ABCD documents stored in the archive file, create the data files, EML document(s) and the archive descriptor(s): [[File:dwcaStartLocal.png]] Depending on the size of your collection (number of records), the amount of information published (number of concepts mapped) and the speed of your server (available memory, number of CPUs/cores), this will take from several seconds to some minutes. For large collections (millions of records), it can take up to hours. After each chunk of 50,000 records processed, a line indicating the progress will be printed, so you will be able to see if the tool is still running. However, independent of the amount of data to be processed, the last lines of the output will look similar to the following: [[File:dwcaEndLocal.png]] When creating an ABCD Archive with the BioCASe Provider Software, it becomes automatically available for download on the BioCASe installation after the process has finished. The DwCA Creator can download the archive directly from this location. So if you're BioCASe installation is accessible from the machine you're running the prototype on, you can spare the trouble of moving the file manually and provide the URL instead of a local file name. To get the download URL for an ABCD Archive you've created, find the archive in question in the list of existing archives in the configuration tool (see [[Archiving#XML_Archives|here]] for more on that). Right-click on the Download link and choose ''Copy link''. Then open a terminal window for the creator tool and provide the link intead of a file name: <pre>./dwca.sh http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command: <pre>dwca.bat http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> ===Output=== The DwCA(s) created can be found in the folder ''output'' created in the base directory (''dwca''). Remember that there will be one DwCA per dataset. So depending upon how many datasets are stored in the source ABCD archive, there will be one or several files in the output folder. On Linux and MacOS, the shell script ''dwca.sh'' will zip all files that make up one DarwinCore Archive into an archive file named after the dataset title. So you could send this files directly to a potential DwCA consumer: [[File:dwcaOutputFolderLinux.png]] On Windows, the batch file ''dwca.bat'' of the prototype doesn't do that zipping. So if you go to the output file after the conversion has finished, you will see several text files and XML documents, all named after the pattern ''<dataset> <filename>''. So for the example above, the output folder would look like this: [[File:dwcaOutputFolderWindows.png]] In order to create a DwCA that can be sent to a consumer, please do the follwing for each ''<dataset>'': # Note down the dataset title, # Remove the dataset title in the file names (''<dataset> <filename>'' becomes ''<filename>''), # Create a zip named ''<dataset>.zip'' with all the files you just renamed. This lack of functionality is due to the limited capabilities of Windows batch files. Once integrated into the Provider Software, you won't need to zip up files manually. ===Trouble Shooting=== If you run into problems when using this tool, please [[Special:Contact|contact us]] for help. The default settings of the transormation should be working for small and medium-sized collections (up to 500,000 records). If the processing is cancelled with an OutOfMemoryError, you should try the follwing: '''Increasing the maximum memory used by the transformer'''<br> Per default, the transformer will use up to 1GB of RAM for the transformation process. In case of an OutOfMemoryError, you should increase this limit to 2GB. Don't be stingy with memory, since it will be only used during the tranformation process and be freed afterwards. Moreover, this is only the upper limit, so actual memory usage might well stay below this limit. To allow the transformer to use more memory, go to the folder ''kettle'' of your installation. Open the file ''kitchen.sh'' (Linux/MacOS) or ''kitchen.bat'' (Windows) in a text editor. Use the ''Find'' feature of the editor to go to the spot in file that specifies the memory limit. Replace the value ''1024'' with the amount of memory you want to allow the transformation to use, so e.g. ''2048'' for 2GB: [[File:dwcaSettingMemory.png]] Save the file and restart the transformation. '''Lowering the number of records stored in memory'''<br> Per default, Kettle will store up to 100,000 records in memory before starting to use temporary files for sorting rows. You can set this threshold to a lower value to save some memory. However, be advised that this will result in heavy disk I/O on your machine, slowing down the transformation. Therefore you should first increase the memory limit before changing this setting. If you decide to do this, just add the desired maximim number of records to be stored in memory as a second parameter when invoking the transformer. So on Linux, type in <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip 10000</pre> in order to set the threshold to 10,000 records. On a Windows machine, type in following: <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip 10000</pre> 3d40d54abc41a194035be1dfce8e323c41e7ae7c 601 600 2012-07-05T10:59:48Z JoergHoletschek 3 /* Usage */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:xmlArchives.png|border|750px]] ===Usage=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create a new Archive</small> allows you to create a new archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:archProcess.png|border]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== XML Archives that use the ABCD 2.06 schema can be converted into DarwinCore Archives. Currently, this feature is only available as a standalone command-line tool. Once thoroughly tested, it will be intregrated into the BioCASe Provider Software and can be trigged from the configuration tool. ===Requirements=== Since this tool is currently a standalone application, it doesn't need to be run on the same machine as the BioCASe Provider Software. But of course it can - if you're running BioCASe on a publicly available server and plan to use DwCAs for production, the BioCASe server is definitely the right place. * Java Runtime Environment 1.5 or later installed * 2+G of RAM available * 100M free disk space * One or several ABCD2.06 Archives (created with BPS3+) ===Installation=== # If you don't have a Java Runtime Environment (version 1.5 or later) installed on your machine, get one [http://www.java.com/de/download/ here]. # Download the DwCA Creator from the [http://www.biocase.org/products/provider_software/index.shtml#DwCA_Creator BioCASe Website] and unzip the archive file to your machine. This will create a folder ''dwca'' for the tool. # Test the installation by opening a terminal (''command prompt'' on Windows) and changing to the ''dwca'' folder. Start the tool by running the batch file ''dwca.bat'' on Windows or the shell script ''dwca.sh'' on Linux/MacOS. Since it doesn't do any processing, the test run should take only a few seconds and print some lines similar to following (you can ignore the warning ''No such file or directory'' in the last line): [[File:dwcaTest.png]] ===Usage=== The converter can be started with the scripts ''dwca.bat'' (on Windows) and ''dwca.sh'' (on Linux/MacOS) and usually takes one argument - the ABCD archive to be converted. The tool accepts both a file stored in the local file system and an archive available for download at a given URL. Let's assume you've created an ABCD2 dump for your collection. You've placed the archive file (named ''AlgaTerra_ABCD_2.06.zip'') in the same folder as the tool (''dwca''). To run the conversion on Linux, open a terminal window, change to the ''dwca'' folder and type in the following: <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command. Note that the base folder for the transformer is the ''kettle'' folder, so you need to specify ''..\'' to direct the transformer to its parent folder for finding the archive file (alternatively, you can put the archive file in the ''kettle'' subfolder and thus spare the ''..\''): <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip</pre> This will unzip the ABCD documents stored in the archive file, create the data files, EML document(s) and the archive descriptor(s): [[File:dwcaStartLocal.png]] Depending on the size of your collection (number of records), the amount of information published (number of concepts mapped) and the speed of your server (available memory, number of CPUs/cores), this will take from several seconds to some minutes. For large collections (millions of records), it can take up to hours. After each chunk of 50,000 records processed, a line indicating the progress will be printed, so you will be able to see if the tool is still running. However, independent of the amount of data to be processed, the last lines of the output will look similar to the following: [[File:dwcaEndLocal.png]] When creating an ABCD Archive with the BioCASe Provider Software, it becomes automatically available for download on the BioCASe installation after the process has finished. The DwCA Creator can download the archive directly from this location. So if you're BioCASe installation is accessible from the machine you're running the prototype on, you can spare the trouble of moving the file manually and provide the URL instead of a local file name. To get the download URL for an ABCD Archive you've created, find the archive in question in the list of existing archives in the configuration tool (see [[Archiving#XML_Archives|here]] for more on that). Right-click on the Download link and choose ''Copy link''. Then open a terminal window for the creator tool and provide the link intead of a file name: <pre>./dwca.sh http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command: <pre>dwca.bat http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> ===Output=== The DwCA(s) created can be found in the folder ''output'' created in the base directory (''dwca''). Remember that there will be one DwCA per dataset. So depending upon how many datasets are stored in the source ABCD archive, there will be one or several files in the output folder. On Linux and MacOS, the shell script ''dwca.sh'' will zip all files that make up one DarwinCore Archive into an archive file named after the dataset title. So you could send this files directly to a potential DwCA consumer: [[File:dwcaOutputFolderLinux.png]] On Windows, the batch file ''dwca.bat'' of the prototype doesn't do that zipping. So if you go to the output file after the conversion has finished, you will see several text files and XML documents, all named after the pattern ''<dataset> <filename>''. So for the example above, the output folder would look like this: [[File:dwcaOutputFolderWindows.png]] In order to create a DwCA that can be sent to a consumer, please do the follwing for each ''<dataset>'': # Note down the dataset title, # Remove the dataset title in the file names (''<dataset> <filename>'' becomes ''<filename>''), # Create a zip named ''<dataset>.zip'' with all the files you just renamed. This lack of functionality is due to the limited capabilities of Windows batch files. Once integrated into the Provider Software, you won't need to zip up files manually. ===Trouble Shooting=== If you run into problems when using this tool, please [[Special:Contact|contact us]] for help. The default settings of the transormation should be working for small and medium-sized collections (up to 500,000 records). If the processing is cancelled with an OutOfMemoryError, you should try the follwing: '''Increasing the maximum memory used by the transformer'''<br> Per default, the transformer will use up to 1GB of RAM for the transformation process. In case of an OutOfMemoryError, you should increase this limit to 2GB. Don't be stingy with memory, since it will be only used during the tranformation process and be freed afterwards. Moreover, this is only the upper limit, so actual memory usage might well stay below this limit. To allow the transformer to use more memory, go to the folder ''kettle'' of your installation. Open the file ''kitchen.sh'' (Linux/MacOS) or ''kitchen.bat'' (Windows) in a text editor. Use the ''Find'' feature of the editor to go to the spot in file that specifies the memory limit. Replace the value ''1024'' with the amount of memory you want to allow the transformation to use, so e.g. ''2048'' for 2GB: [[File:dwcaSettingMemory.png]] Save the file and restart the transformation. '''Lowering the number of records stored in memory'''<br> Per default, Kettle will store up to 100,000 records in memory before starting to use temporary files for sorting rows. You can set this threshold to a lower value to save some memory. However, be advised that this will result in heavy disk I/O on your machine, slowing down the transformation. Therefore you should first increase the memory limit before changing this setting. If you decide to do this, just add the desired maximim number of records to be stored in memory as a second parameter when invoking the transformer. So on Linux, type in <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip 10000</pre> in order to set the threshold to 10,000 records. On a Windows machine, type in following: <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip 10000</pre> fee12dcdddbca5099847603a8a43b0541d5a285e 602 601 2012-07-05T11:06:40Z JoergHoletschek 3 /* Usage */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:xmlArchives.png|border|750px]] ===Usage=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== XML Archives that use the ABCD 2.06 schema can be converted into DarwinCore Archives. Currently, this feature is only available as a standalone command-line tool. Once thoroughly tested, it will be intregrated into the BioCASe Provider Software and can be trigged from the configuration tool. ===Requirements=== Since this tool is currently a standalone application, it doesn't need to be run on the same machine as the BioCASe Provider Software. But of course it can - if you're running BioCASe on a publicly available server and plan to use DwCAs for production, the BioCASe server is definitely the right place. * Java Runtime Environment 1.5 or later installed * 2+G of RAM available * 100M free disk space * One or several ABCD2.06 Archives (created with BPS3+) ===Installation=== # If you don't have a Java Runtime Environment (version 1.5 or later) installed on your machine, get one [http://www.java.com/de/download/ here]. # Download the DwCA Creator from the [http://www.biocase.org/products/provider_software/index.shtml#DwCA_Creator BioCASe Website] and unzip the archive file to your machine. This will create a folder ''dwca'' for the tool. # Test the installation by opening a terminal (''command prompt'' on Windows) and changing to the ''dwca'' folder. Start the tool by running the batch file ''dwca.bat'' on Windows or the shell script ''dwca.sh'' on Linux/MacOS. Since it doesn't do any processing, the test run should take only a few seconds and print some lines similar to following (you can ignore the warning ''No such file or directory'' in the last line): [[File:dwcaTest.png]] ===Usage=== The converter can be started with the scripts ''dwca.bat'' (on Windows) and ''dwca.sh'' (on Linux/MacOS) and usually takes one argument - the ABCD archive to be converted. The tool accepts both a file stored in the local file system and an archive available for download at a given URL. Let's assume you've created an ABCD2 dump for your collection. You've placed the archive file (named ''AlgaTerra_ABCD_2.06.zip'') in the same folder as the tool (''dwca''). To run the conversion on Linux, open a terminal window, change to the ''dwca'' folder and type in the following: <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command. Note that the base folder for the transformer is the ''kettle'' folder, so you need to specify ''..\'' to direct the transformer to its parent folder for finding the archive file (alternatively, you can put the archive file in the ''kettle'' subfolder and thus spare the ''..\''): <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip</pre> This will unzip the ABCD documents stored in the archive file, create the data files, EML document(s) and the archive descriptor(s): [[File:dwcaStartLocal.png]] Depending on the size of your collection (number of records), the amount of information published (number of concepts mapped) and the speed of your server (available memory, number of CPUs/cores), this will take from several seconds to some minutes. For large collections (millions of records), it can take up to hours. After each chunk of 50,000 records processed, a line indicating the progress will be printed, so you will be able to see if the tool is still running. However, independent of the amount of data to be processed, the last lines of the output will look similar to the following: [[File:dwcaEndLocal.png]] When creating an ABCD Archive with the BioCASe Provider Software, it becomes automatically available for download on the BioCASe installation after the process has finished. The DwCA Creator can download the archive directly from this location. So if you're BioCASe installation is accessible from the machine you're running the prototype on, you can spare the trouble of moving the file manually and provide the URL instead of a local file name. To get the download URL for an ABCD Archive you've created, find the archive in question in the list of existing archives in the configuration tool (see [[Archiving#XML_Archives|here]] for more on that). Right-click on the Download link and choose ''Copy link''. Then open a terminal window for the creator tool and provide the link intead of a file name: <pre>./dwca.sh http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command: <pre>dwca.bat http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> ===Output=== The DwCA(s) created can be found in the folder ''output'' created in the base directory (''dwca''). Remember that there will be one DwCA per dataset. So depending upon how many datasets are stored in the source ABCD archive, there will be one or several files in the output folder. On Linux and MacOS, the shell script ''dwca.sh'' will zip all files that make up one DarwinCore Archive into an archive file named after the dataset title. So you could send this files directly to a potential DwCA consumer: [[File:dwcaOutputFolderLinux.png]] On Windows, the batch file ''dwca.bat'' of the prototype doesn't do that zipping. So if you go to the output file after the conversion has finished, you will see several text files and XML documents, all named after the pattern ''<dataset> <filename>''. So for the example above, the output folder would look like this: [[File:dwcaOutputFolderWindows.png]] In order to create a DwCA that can be sent to a consumer, please do the follwing for each ''<dataset>'': # Note down the dataset title, # Remove the dataset title in the file names (''<dataset> <filename>'' becomes ''<filename>''), # Create a zip named ''<dataset>.zip'' with all the files you just renamed. This lack of functionality is due to the limited capabilities of Windows batch files. Once integrated into the Provider Software, you won't need to zip up files manually. ===Trouble Shooting=== If you run into problems when using this tool, please [[Special:Contact|contact us]] for help. The default settings of the transormation should be working for small and medium-sized collections (up to 500,000 records). If the processing is cancelled with an OutOfMemoryError, you should try the follwing: '''Increasing the maximum memory used by the transformer'''<br> Per default, the transformer will use up to 1GB of RAM for the transformation process. In case of an OutOfMemoryError, you should increase this limit to 2GB. Don't be stingy with memory, since it will be only used during the tranformation process and be freed afterwards. Moreover, this is only the upper limit, so actual memory usage might well stay below this limit. To allow the transformer to use more memory, go to the folder ''kettle'' of your installation. Open the file ''kitchen.sh'' (Linux/MacOS) or ''kitchen.bat'' (Windows) in a text editor. Use the ''Find'' feature of the editor to go to the spot in file that specifies the memory limit. Replace the value ''1024'' with the amount of memory you want to allow the transformation to use, so e.g. ''2048'' for 2GB: [[File:dwcaSettingMemory.png]] Save the file and restart the transformation. '''Lowering the number of records stored in memory'''<br> Per default, Kettle will store up to 100,000 records in memory before starting to use temporary files for sorting rows. You can set this threshold to a lower value to save some memory. However, be advised that this will result in heavy disk I/O on your machine, slowing down the transformation. Therefore you should first increase the memory limit before changing this setting. If you decide to do this, just add the desired maximim number of records to be stored in memory as a second parameter when invoking the transformer. So on Linux, type in <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip 10000</pre> in order to set the threshold to 10,000 records. On a Windows machine, type in following: <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip 10000</pre> 5721a7863d390c98871e313495c84954e97a639b 604 602 2012-07-05T11:07:57Z JoergHoletschek 3 /* Usage */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:xmlArchives.png|border|750px]] ===Usage=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|775px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== XML Archives that use the ABCD 2.06 schema can be converted into DarwinCore Archives. Currently, this feature is only available as a standalone command-line tool. Once thoroughly tested, it will be intregrated into the BioCASe Provider Software and can be trigged from the configuration tool. ===Requirements=== Since this tool is currently a standalone application, it doesn't need to be run on the same machine as the BioCASe Provider Software. But of course it can - if you're running BioCASe on a publicly available server and plan to use DwCAs for production, the BioCASe server is definitely the right place. * Java Runtime Environment 1.5 or later installed * 2+G of RAM available * 100M free disk space * One or several ABCD2.06 Archives (created with BPS3+) ===Installation=== # If you don't have a Java Runtime Environment (version 1.5 or later) installed on your machine, get one [http://www.java.com/de/download/ here]. # Download the DwCA Creator from the [http://www.biocase.org/products/provider_software/index.shtml#DwCA_Creator BioCASe Website] and unzip the archive file to your machine. This will create a folder ''dwca'' for the tool. # Test the installation by opening a terminal (''command prompt'' on Windows) and changing to the ''dwca'' folder. Start the tool by running the batch file ''dwca.bat'' on Windows or the shell script ''dwca.sh'' on Linux/MacOS. Since it doesn't do any processing, the test run should take only a few seconds and print some lines similar to following (you can ignore the warning ''No such file or directory'' in the last line): [[File:dwcaTest.png]] ===Usage=== The converter can be started with the scripts ''dwca.bat'' (on Windows) and ''dwca.sh'' (on Linux/MacOS) and usually takes one argument - the ABCD archive to be converted. The tool accepts both a file stored in the local file system and an archive available for download at a given URL. Let's assume you've created an ABCD2 dump for your collection. You've placed the archive file (named ''AlgaTerra_ABCD_2.06.zip'') in the same folder as the tool (''dwca''). To run the conversion on Linux, open a terminal window, change to the ''dwca'' folder and type in the following: <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command. Note that the base folder for the transformer is the ''kettle'' folder, so you need to specify ''..\'' to direct the transformer to its parent folder for finding the archive file (alternatively, you can put the archive file in the ''kettle'' subfolder and thus spare the ''..\''): <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip</pre> This will unzip the ABCD documents stored in the archive file, create the data files, EML document(s) and the archive descriptor(s): [[File:dwcaStartLocal.png]] Depending on the size of your collection (number of records), the amount of information published (number of concepts mapped) and the speed of your server (available memory, number of CPUs/cores), this will take from several seconds to some minutes. For large collections (millions of records), it can take up to hours. After each chunk of 50,000 records processed, a line indicating the progress will be printed, so you will be able to see if the tool is still running. However, independent of the amount of data to be processed, the last lines of the output will look similar to the following: [[File:dwcaEndLocal.png]] When creating an ABCD Archive with the BioCASe Provider Software, it becomes automatically available for download on the BioCASe installation after the process has finished. The DwCA Creator can download the archive directly from this location. So if you're BioCASe installation is accessible from the machine you're running the prototype on, you can spare the trouble of moving the file manually and provide the URL instead of a local file name. To get the download URL for an ABCD Archive you've created, find the archive in question in the list of existing archives in the configuration tool (see [[Archiving#XML_Archives|here]] for more on that). Right-click on the Download link and choose ''Copy link''. Then open a terminal window for the creator tool and provide the link intead of a file name: <pre>./dwca.sh http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command: <pre>dwca.bat http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> ===Output=== The DwCA(s) created can be found in the folder ''output'' created in the base directory (''dwca''). Remember that there will be one DwCA per dataset. So depending upon how many datasets are stored in the source ABCD archive, there will be one or several files in the output folder. On Linux and MacOS, the shell script ''dwca.sh'' will zip all files that make up one DarwinCore Archive into an archive file named after the dataset title. So you could send this files directly to a potential DwCA consumer: [[File:dwcaOutputFolderLinux.png]] On Windows, the batch file ''dwca.bat'' of the prototype doesn't do that zipping. So if you go to the output file after the conversion has finished, you will see several text files and XML documents, all named after the pattern ''<dataset> <filename>''. So for the example above, the output folder would look like this: [[File:dwcaOutputFolderWindows.png]] In order to create a DwCA that can be sent to a consumer, please do the follwing for each ''<dataset>'': # Note down the dataset title, # Remove the dataset title in the file names (''<dataset> <filename>'' becomes ''<filename>''), # Create a zip named ''<dataset>.zip'' with all the files you just renamed. This lack of functionality is due to the limited capabilities of Windows batch files. Once integrated into the Provider Software, you won't need to zip up files manually. ===Trouble Shooting=== If you run into problems when using this tool, please [[Special:Contact|contact us]] for help. The default settings of the transormation should be working for small and medium-sized collections (up to 500,000 records). If the processing is cancelled with an OutOfMemoryError, you should try the follwing: '''Increasing the maximum memory used by the transformer'''<br> Per default, the transformer will use up to 1GB of RAM for the transformation process. In case of an OutOfMemoryError, you should increase this limit to 2GB. Don't be stingy with memory, since it will be only used during the tranformation process and be freed afterwards. Moreover, this is only the upper limit, so actual memory usage might well stay below this limit. To allow the transformer to use more memory, go to the folder ''kettle'' of your installation. Open the file ''kitchen.sh'' (Linux/MacOS) or ''kitchen.bat'' (Windows) in a text editor. Use the ''Find'' feature of the editor to go to the spot in file that specifies the memory limit. Replace the value ''1024'' with the amount of memory you want to allow the transformation to use, so e.g. ''2048'' for 2GB: [[File:dwcaSettingMemory.png]] Save the file and restart the transformation. '''Lowering the number of records stored in memory'''<br> Per default, Kettle will store up to 100,000 records in memory before starting to use temporary files for sorting rows. You can set this threshold to a lower value to save some memory. However, be advised that this will result in heavy disk I/O on your machine, slowing down the transformation. Therefore you should first increase the memory limit before changing this setting. If you decide to do this, just add the desired maximim number of records to be stored in memory as a second parameter when invoking the transformer. So on Linux, type in <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip 10000</pre> in order to set the threshold to 10,000 records. On a Windows machine, type in following: <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip 10000</pre> e3fa540c53cb736cfa5963f700b505a1c2fb3005 605 604 2012-07-05T11:25:19Z JoergHoletschek 3 /* Usage */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:xmlArchives.png|border|750px]] ===Usage=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|775px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== XML Archives that use the ABCD 2.06 schema can be converted into DarwinCore Archives. Currently, this feature is only available as a standalone command-line tool. Once thoroughly tested, it will be intregrated into the BioCASe Provider Software and can be trigged from the configuration tool. ===Requirements=== Since this tool is currently a standalone application, it doesn't need to be run on the same machine as the BioCASe Provider Software. But of course it can - if you're running BioCASe on a publicly available server and plan to use DwCAs for production, the BioCASe server is definitely the right place. * Java Runtime Environment 1.5 or later installed * 2+G of RAM available * 100M free disk space * One or several ABCD2.06 Archives (created with BPS3+) ===Installation=== # If you don't have a Java Runtime Environment (version 1.5 or later) installed on your machine, get one [http://www.java.com/de/download/ here]. # Download the DwCA Creator from the [http://www.biocase.org/products/provider_software/index.shtml#DwCA_Creator BioCASe Website] and unzip the archive file to your machine. This will create a folder ''dwca'' for the tool. # Test the installation by opening a terminal (''command prompt'' on Windows) and changing to the ''dwca'' folder. Start the tool by running the batch file ''dwca.bat'' on Windows or the shell script ''dwca.sh'' on Linux/MacOS. Since it doesn't do any processing, the test run should take only a few seconds and print some lines similar to following (you can ignore the warning ''No such file or directory'' in the last line): [[File:dwcaTest.png]] ===Usage=== The converter can be started with the scripts ''dwca.bat'' (on Windows) and ''dwca.sh'' (on Linux/MacOS) and usually takes one argument - the ABCD archive to be converted. The tool accepts both a file stored in the local file system and an archive available for download at a given URL. Let's assume you've created an ABCD2 dump for your collection. You've placed the archive file (named ''AlgaTerra_ABCD_2.06.zip'') in the same folder as the tool (''dwca''). To run the conversion on Linux, open a terminal window, change to the ''dwca'' folder and type in the following: <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command. Note that the base folder for the transformer is the ''kettle'' folder, so you need to specify ''..\'' to direct the transformer to its parent folder for finding the archive file (alternatively, you can put the archive file in the ''kettle'' subfolder and thus spare the ''..\''): <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip</pre> This will unzip the ABCD documents stored in the archive file, create the data files, EML document(s) and the archive descriptor(s): [[File:dwcaStartLocal.png]] Depending on the size of your collection (number of records), the amount of information published (number of concepts mapped) and the speed of your server (available memory, number of CPUs/cores), this will take from several seconds to some minutes. For large collections (millions of records), it can take up to hours. After each chunk of 50,000 records processed, a line indicating the progress will be printed, so you will be able to see if the tool is still running. However, independent of the amount of data to be processed, the last lines of the output will look similar to the following: [[File:dwcaEndLocal.png]] When creating an ABCD Archive with the BioCASe Provider Software, it becomes automatically available for download on the BioCASe installation after the process has finished. The DwCA Creator can download the archive directly from this location. So if you're BioCASe installation is accessible from the machine you're running the prototype on, you can spare the trouble of moving the file manually and provide the URL instead of a local file name. To get the download URL for an ABCD Archive you've created, find the archive in question in the list of existing archives in the configuration tool (see [[Archiving#XML_Archives|here]] for more on that). Right-click on the Download link and choose ''Copy link''. Then open a terminal window for the creator tool and provide the link intead of a file name: <pre>./dwca.sh http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command: <pre>dwca.bat http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> ===Output=== The DwCA(s) created can be found in the folder ''output'' created in the base directory (''dwca''). Remember that there will be one DwCA per dataset. So depending upon how many datasets are stored in the source ABCD archive, there will be one or several files in the output folder. On Linux and MacOS, the shell script ''dwca.sh'' will zip all files that make up one DarwinCore Archive into an archive file named after the dataset title. So you could send this files directly to a potential DwCA consumer: [[File:dwcaOutputFolderLinux.png]] On Windows, the batch file ''dwca.bat'' of the prototype doesn't do that zipping. So if you go to the output file after the conversion has finished, you will see several text files and XML documents, all named after the pattern ''<dataset> <filename>''. So for the example above, the output folder would look like this: [[File:dwcaOutputFolderWindows.png]] In order to create a DwCA that can be sent to a consumer, please do the follwing for each ''<dataset>'': # Note down the dataset title, # Remove the dataset title in the file names (''<dataset> <filename>'' becomes ''<filename>''), # Create a zip named ''<dataset>.zip'' with all the files you just renamed. This lack of functionality is due to the limited capabilities of Windows batch files. Once integrated into the Provider Software, you won't need to zip up files manually. ===Trouble Shooting=== If you run into problems when using this tool, please [[Special:Contact|contact us]] for help. The default settings of the transormation should be working for small and medium-sized collections (up to 500,000 records). If the processing is cancelled with an OutOfMemoryError, you should try the follwing: '''Increasing the maximum memory used by the transformer'''<br> Per default, the transformer will use up to 1GB of RAM for the transformation process. In case of an OutOfMemoryError, you should increase this limit to 2GB. Don't be stingy with memory, since it will be only used during the tranformation process and be freed afterwards. Moreover, this is only the upper limit, so actual memory usage might well stay below this limit. To allow the transformer to use more memory, go to the folder ''kettle'' of your installation. Open the file ''kitchen.sh'' (Linux/MacOS) or ''kitchen.bat'' (Windows) in a text editor. Use the ''Find'' feature of the editor to go to the spot in file that specifies the memory limit. Replace the value ''1024'' with the amount of memory you want to allow the transformation to use, so e.g. ''2048'' for 2GB: [[File:dwcaSettingMemory.png]] Save the file and restart the transformation. '''Lowering the number of records stored in memory'''<br> Per default, Kettle will store up to 100,000 records in memory before starting to use temporary files for sorting rows. You can set this threshold to a lower value to save some memory. However, be advised that this will result in heavy disk I/O on your machine, slowing down the transformation. Therefore you should first increase the memory limit before changing this setting. If you decide to do this, just add the desired maximim number of records to be stored in memory as a second parameter when invoking the transformer. So on Linux, type in <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip 10000</pre> in order to set the threshold to 10,000 records. On a Windows machine, type in following: <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip 10000</pre> ff8c344dc5ce343ff7c460cda25530adca061e15 606 605 2012-07-05T11:27:27Z JoergHoletschek 3 /* Customizing the Archive */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:xmlArchives.png|border|750px]] ===Usage=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|775px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== XML Archives that use the ABCD 2.06 schema can be converted into DarwinCore Archives. Currently, this feature is only available as a standalone command-line tool. Once thoroughly tested, it will be intregrated into the BioCASe Provider Software and can be trigged from the configuration tool. ===Requirements=== Since this tool is currently a standalone application, it doesn't need to be run on the same machine as the BioCASe Provider Software. But of course it can - if you're running BioCASe on a publicly available server and plan to use DwCAs for production, the BioCASe server is definitely the right place. * Java Runtime Environment 1.5 or later installed * 2+G of RAM available * 100M free disk space * One or several ABCD2.06 Archives (created with BPS3+) ===Installation=== # If you don't have a Java Runtime Environment (version 1.5 or later) installed on your machine, get one [http://www.java.com/de/download/ here]. # Download the DwCA Creator from the [http://www.biocase.org/products/provider_software/index.shtml#DwCA_Creator BioCASe Website] and unzip the archive file to your machine. This will create a folder ''dwca'' for the tool. # Test the installation by opening a terminal (''command prompt'' on Windows) and changing to the ''dwca'' folder. Start the tool by running the batch file ''dwca.bat'' on Windows or the shell script ''dwca.sh'' on Linux/MacOS. Since it doesn't do any processing, the test run should take only a few seconds and print some lines similar to following (you can ignore the warning ''No such file or directory'' in the last line): [[File:dwcaTest.png]] ===Usage=== The converter can be started with the scripts ''dwca.bat'' (on Windows) and ''dwca.sh'' (on Linux/MacOS) and usually takes one argument - the ABCD archive to be converted. The tool accepts both a file stored in the local file system and an archive available for download at a given URL. Let's assume you've created an ABCD2 dump for your collection. You've placed the archive file (named ''AlgaTerra_ABCD_2.06.zip'') in the same folder as the tool (''dwca''). To run the conversion on Linux, open a terminal window, change to the ''dwca'' folder and type in the following: <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command. Note that the base folder for the transformer is the ''kettle'' folder, so you need to specify ''..\'' to direct the transformer to its parent folder for finding the archive file (alternatively, you can put the archive file in the ''kettle'' subfolder and thus spare the ''..\''): <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip</pre> This will unzip the ABCD documents stored in the archive file, create the data files, EML document(s) and the archive descriptor(s): [[File:dwcaStartLocal.png]] Depending on the size of your collection (number of records), the amount of information published (number of concepts mapped) and the speed of your server (available memory, number of CPUs/cores), this will take from several seconds to some minutes. For large collections (millions of records), it can take up to hours. After each chunk of 50,000 records processed, a line indicating the progress will be printed, so you will be able to see if the tool is still running. However, independent of the amount of data to be processed, the last lines of the output will look similar to the following: [[File:dwcaEndLocal.png]] When creating an ABCD Archive with the BioCASe Provider Software, it becomes automatically available for download on the BioCASe installation after the process has finished. The DwCA Creator can download the archive directly from this location. So if you're BioCASe installation is accessible from the machine you're running the prototype on, you can spare the trouble of moving the file manually and provide the URL instead of a local file name. To get the download URL for an ABCD Archive you've created, find the archive in question in the list of existing archives in the configuration tool (see [[Archiving#XML_Archives|here]] for more on that). Right-click on the Download link and choose ''Copy link''. Then open a terminal window for the creator tool and provide the link intead of a file name: <pre>./dwca.sh http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command: <pre>dwca.bat http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> ===Output=== The DwCA(s) created can be found in the folder ''output'' created in the base directory (''dwca''). Remember that there will be one DwCA per dataset. So depending upon how many datasets are stored in the source ABCD archive, there will be one or several files in the output folder. On Linux and MacOS, the shell script ''dwca.sh'' will zip all files that make up one DarwinCore Archive into an archive file named after the dataset title. So you could send this files directly to a potential DwCA consumer: [[File:dwcaOutputFolderLinux.png]] On Windows, the batch file ''dwca.bat'' of the prototype doesn't do that zipping. So if you go to the output file after the conversion has finished, you will see several text files and XML documents, all named after the pattern ''<dataset> <filename>''. So for the example above, the output folder would look like this: [[File:dwcaOutputFolderWindows.png]] In order to create a DwCA that can be sent to a consumer, please do the follwing for each ''<dataset>'': # Note down the dataset title, # Remove the dataset title in the file names (''<dataset> <filename>'' becomes ''<filename>''), # Create a zip named ''<dataset>.zip'' with all the files you just renamed. This lack of functionality is due to the limited capabilities of Windows batch files. Once integrated into the Provider Software, you won't need to zip up files manually. ===Trouble Shooting=== If you run into problems when using this tool, please [[Special:Contact|contact us]] for help. The default settings of the transormation should be working for small and medium-sized collections (up to 500,000 records). If the processing is cancelled with an OutOfMemoryError, you should try the follwing: '''Increasing the maximum memory used by the transformer'''<br> Per default, the transformer will use up to 1GB of RAM for the transformation process. In case of an OutOfMemoryError, you should increase this limit to 2GB. Don't be stingy with memory, since it will be only used during the tranformation process and be freed afterwards. Moreover, this is only the upper limit, so actual memory usage might well stay below this limit. To allow the transformer to use more memory, go to the folder ''kettle'' of your installation. Open the file ''kitchen.sh'' (Linux/MacOS) or ''kitchen.bat'' (Windows) in a text editor. Use the ''Find'' feature of the editor to go to the spot in file that specifies the memory limit. Replace the value ''1024'' with the amount of memory you want to allow the transformation to use, so e.g. ''2048'' for 2GB: [[File:dwcaSettingMemory.png]] Save the file and restart the transformation. '''Lowering the number of records stored in memory'''<br> Per default, Kettle will store up to 100,000 records in memory before starting to use temporary files for sorting rows. You can set this threshold to a lower value to save some memory. However, be advised that this will result in heavy disk I/O on your machine, slowing down the transformation. Therefore you should first increase the memory limit before changing this setting. If you decide to do this, just add the desired maximim number of records to be stored in memory as a second parameter when invoking the transformer. So on Linux, type in <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip 10000</pre> in order to set the threshold to 10,000 records. On a Windows machine, type in following: <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip 10000</pre> 1e4f7905d2f41c7202a3f70c1d37e49bc16e67f7 607 606 2012-07-05T11:29:43Z JoergHoletschek 3 /* XML Archives */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Usage=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|775px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== XML Archives that use the ABCD 2.06 schema can be converted into DarwinCore Archives. Currently, this feature is only available as a standalone command-line tool. Once thoroughly tested, it will be intregrated into the BioCASe Provider Software and can be trigged from the configuration tool. ===Requirements=== Since this tool is currently a standalone application, it doesn't need to be run on the same machine as the BioCASe Provider Software. But of course it can - if you're running BioCASe on a publicly available server and plan to use DwCAs for production, the BioCASe server is definitely the right place. * Java Runtime Environment 1.5 or later installed * 2+G of RAM available * 100M free disk space * One or several ABCD2.06 Archives (created with BPS3+) ===Installation=== # If you don't have a Java Runtime Environment (version 1.5 or later) installed on your machine, get one [http://www.java.com/de/download/ here]. # Download the DwCA Creator from the [http://www.biocase.org/products/provider_software/index.shtml#DwCA_Creator BioCASe Website] and unzip the archive file to your machine. This will create a folder ''dwca'' for the tool. # Test the installation by opening a terminal (''command prompt'' on Windows) and changing to the ''dwca'' folder. Start the tool by running the batch file ''dwca.bat'' on Windows or the shell script ''dwca.sh'' on Linux/MacOS. Since it doesn't do any processing, the test run should take only a few seconds and print some lines similar to following (you can ignore the warning ''No such file or directory'' in the last line): [[File:dwcaTest.png]] ===Usage=== The converter can be started with the scripts ''dwca.bat'' (on Windows) and ''dwca.sh'' (on Linux/MacOS) and usually takes one argument - the ABCD archive to be converted. The tool accepts both a file stored in the local file system and an archive available for download at a given URL. Let's assume you've created an ABCD2 dump for your collection. You've placed the archive file (named ''AlgaTerra_ABCD_2.06.zip'') in the same folder as the tool (''dwca''). To run the conversion on Linux, open a terminal window, change to the ''dwca'' folder and type in the following: <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command. Note that the base folder for the transformer is the ''kettle'' folder, so you need to specify ''..\'' to direct the transformer to its parent folder for finding the archive file (alternatively, you can put the archive file in the ''kettle'' subfolder and thus spare the ''..\''): <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip</pre> This will unzip the ABCD documents stored in the archive file, create the data files, EML document(s) and the archive descriptor(s): [[File:dwcaStartLocal.png]] Depending on the size of your collection (number of records), the amount of information published (number of concepts mapped) and the speed of your server (available memory, number of CPUs/cores), this will take from several seconds to some minutes. For large collections (millions of records), it can take up to hours. After each chunk of 50,000 records processed, a line indicating the progress will be printed, so you will be able to see if the tool is still running. However, independent of the amount of data to be processed, the last lines of the output will look similar to the following: [[File:dwcaEndLocal.png]] When creating an ABCD Archive with the BioCASe Provider Software, it becomes automatically available for download on the BioCASe installation after the process has finished. The DwCA Creator can download the archive directly from this location. So if you're BioCASe installation is accessible from the machine you're running the prototype on, you can spare the trouble of moving the file manually and provide the URL instead of a local file name. To get the download URL for an ABCD Archive you've created, find the archive in question in the list of existing archives in the configuration tool (see [[Archiving#XML_Archives|here]] for more on that). Right-click on the Download link and choose ''Copy link''. Then open a terminal window for the creator tool and provide the link intead of a file name: <pre>./dwca.sh http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command: <pre>dwca.bat http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> ===Output=== The DwCA(s) created can be found in the folder ''output'' created in the base directory (''dwca''). Remember that there will be one DwCA per dataset. So depending upon how many datasets are stored in the source ABCD archive, there will be one or several files in the output folder. On Linux and MacOS, the shell script ''dwca.sh'' will zip all files that make up one DarwinCore Archive into an archive file named after the dataset title. So you could send this files directly to a potential DwCA consumer: [[File:dwcaOutputFolderLinux.png]] On Windows, the batch file ''dwca.bat'' of the prototype doesn't do that zipping. So if you go to the output file after the conversion has finished, you will see several text files and XML documents, all named after the pattern ''<dataset> <filename>''. So for the example above, the output folder would look like this: [[File:dwcaOutputFolderWindows.png]] In order to create a DwCA that can be sent to a consumer, please do the follwing for each ''<dataset>'': # Note down the dataset title, # Remove the dataset title in the file names (''<dataset> <filename>'' becomes ''<filename>''), # Create a zip named ''<dataset>.zip'' with all the files you just renamed. This lack of functionality is due to the limited capabilities of Windows batch files. Once integrated into the Provider Software, you won't need to zip up files manually. ===Trouble Shooting=== If you run into problems when using this tool, please [[Special:Contact|contact us]] for help. The default settings of the transormation should be working for small and medium-sized collections (up to 500,000 records). If the processing is cancelled with an OutOfMemoryError, you should try the follwing: '''Increasing the maximum memory used by the transformer'''<br> Per default, the transformer will use up to 1GB of RAM for the transformation process. In case of an OutOfMemoryError, you should increase this limit to 2GB. Don't be stingy with memory, since it will be only used during the tranformation process and be freed afterwards. Moreover, this is only the upper limit, so actual memory usage might well stay below this limit. To allow the transformer to use more memory, go to the folder ''kettle'' of your installation. Open the file ''kitchen.sh'' (Linux/MacOS) or ''kitchen.bat'' (Windows) in a text editor. Use the ''Find'' feature of the editor to go to the spot in file that specifies the memory limit. Replace the value ''1024'' with the amount of memory you want to allow the transformation to use, so e.g. ''2048'' for 2GB: [[File:dwcaSettingMemory.png]] Save the file and restart the transformation. '''Lowering the number of records stored in memory'''<br> Per default, Kettle will store up to 100,000 records in memory before starting to use temporary files for sorting rows. You can set this threshold to a lower value to save some memory. However, be advised that this will result in heavy disk I/O on your machine, slowing down the transformation. Therefore you should first increase the memory limit before changing this setting. If you decide to do this, just add the desired maximim number of records to be stored in memory as a second parameter when invoking the transformer. So on Linux, type in <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip 10000</pre> in order to set the threshold to 10,000 records. On a Windows machine, type in following: <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip 10000</pre> 25955bbc7277393cf74e924947a677590361e29e 608 607 2012-07-05T11:51:08Z JoergHoletschek 3 /* DarwinCore Archives */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Usage=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|775px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. XML Archives that use the ABCD 2.06 schema can be converted into DarwinCore Archives. Currently, this feature is only available as a standalone command-line tool. Once thoroughly tested, it will be intregrated into the BioCASe Provider Software and can be trigged from the configuration tool. ===Requirements=== Since this tool is currently a standalone application, it doesn't need to be run on the same machine as the BioCASe Provider Software. But of course it can - if you're running BioCASe on a publicly available server and plan to use DwCAs for production, the BioCASe server is definitely the right place. * Java Runtime Environment 1.5 or later installed * 2+G of RAM available * 100M free disk space * One or several ABCD2.06 Archives (created with BPS3+) ===Installation=== # If you don't have a Java Runtime Environment (version 1.5 or later) installed on your machine, get one [http://www.java.com/de/download/ here]. # Download the DwCA Creator from the [http://www.biocase.org/products/provider_software/index.shtml#DwCA_Creator BioCASe Website] and unzip the archive file to your machine. This will create a folder ''dwca'' for the tool. # Test the installation by opening a terminal (''command prompt'' on Windows) and changing to the ''dwca'' folder. Start the tool by running the batch file ''dwca.bat'' on Windows or the shell script ''dwca.sh'' on Linux/MacOS. Since it doesn't do any processing, the test run should take only a few seconds and print some lines similar to following (you can ignore the warning ''No such file or directory'' in the last line): [[File:dwcaTest.png]] ===Usage=== The converter can be started with the scripts ''dwca.bat'' (on Windows) and ''dwca.sh'' (on Linux/MacOS) and usually takes one argument - the ABCD archive to be converted. The tool accepts both a file stored in the local file system and an archive available for download at a given URL. Let's assume you've created an ABCD2 dump for your collection. You've placed the archive file (named ''AlgaTerra_ABCD_2.06.zip'') in the same folder as the tool (''dwca''). To run the conversion on Linux, open a terminal window, change to the ''dwca'' folder and type in the following: <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command. Note that the base folder for the transformer is the ''kettle'' folder, so you need to specify ''..\'' to direct the transformer to its parent folder for finding the archive file (alternatively, you can put the archive file in the ''kettle'' subfolder and thus spare the ''..\''): <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip</pre> This will unzip the ABCD documents stored in the archive file, create the data files, EML document(s) and the archive descriptor(s): [[File:dwcaStartLocal.png]] Depending on the size of your collection (number of records), the amount of information published (number of concepts mapped) and the speed of your server (available memory, number of CPUs/cores), this will take from several seconds to some minutes. For large collections (millions of records), it can take up to hours. After each chunk of 50,000 records processed, a line indicating the progress will be printed, so you will be able to see if the tool is still running. However, independent of the amount of data to be processed, the last lines of the output will look similar to the following: [[File:dwcaEndLocal.png]] When creating an ABCD Archive with the BioCASe Provider Software, it becomes automatically available for download on the BioCASe installation after the process has finished. The DwCA Creator can download the archive directly from this location. So if you're BioCASe installation is accessible from the machine you're running the prototype on, you can spare the trouble of moving the file manually and provide the URL instead of a local file name. To get the download URL for an ABCD Archive you've created, find the archive in question in the list of existing archives in the configuration tool (see [[Archiving#XML_Archives|here]] for more on that). Right-click on the Download link and choose ''Copy link''. Then open a terminal window for the creator tool and provide the link intead of a file name: <pre>./dwca.sh http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command: <pre>dwca.bat http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> ===Output=== The DwCA(s) created can be found in the folder ''output'' created in the base directory (''dwca''). Remember that there will be one DwCA per dataset. So depending upon how many datasets are stored in the source ABCD archive, there will be one or several files in the output folder. On Linux and MacOS, the shell script ''dwca.sh'' will zip all files that make up one DarwinCore Archive into an archive file named after the dataset title. So you could send this files directly to a potential DwCA consumer: [[File:dwcaOutputFolderLinux.png]] On Windows, the batch file ''dwca.bat'' of the prototype doesn't do that zipping. So if you go to the output file after the conversion has finished, you will see several text files and XML documents, all named after the pattern ''<dataset> <filename>''. So for the example above, the output folder would look like this: [[File:dwcaOutputFolderWindows.png]] In order to create a DwCA that can be sent to a consumer, please do the follwing for each ''<dataset>'': # Note down the dataset title, # Remove the dataset title in the file names (''<dataset> <filename>'' becomes ''<filename>''), # Create a zip named ''<dataset>.zip'' with all the files you just renamed. This lack of functionality is due to the limited capabilities of Windows batch files. Once integrated into the Provider Software, you won't need to zip up files manually. ===Trouble Shooting=== If you run into problems when using this tool, please [[Special:Contact|contact us]] for help. The default settings of the transormation should be working for small and medium-sized collections (up to 500,000 records). If the processing is cancelled with an OutOfMemoryError, you should try the follwing: '''Increasing the maximum memory used by the transformer'''<br> Per default, the transformer will use up to 1GB of RAM for the transformation process. In case of an OutOfMemoryError, you should increase this limit to 2GB. Don't be stingy with memory, since it will be only used during the tranformation process and be freed afterwards. Moreover, this is only the upper limit, so actual memory usage might well stay below this limit. To allow the transformer to use more memory, go to the folder ''kettle'' of your installation. Open the file ''kitchen.sh'' (Linux/MacOS) or ''kitchen.bat'' (Windows) in a text editor. Use the ''Find'' feature of the editor to go to the spot in file that specifies the memory limit. Replace the value ''1024'' with the amount of memory you want to allow the transformation to use, so e.g. ''2048'' for 2GB: [[File:dwcaSettingMemory.png]] Save the file and restart the transformation. '''Lowering the number of records stored in memory'''<br> Per default, Kettle will store up to 100,000 records in memory before starting to use temporary files for sorting rows. You can set this threshold to a lower value to save some memory. However, be advised that this will result in heavy disk I/O on your machine, slowing down the transformation. Therefore you should first increase the memory limit before changing this setting. If you decide to do this, just add the desired maximim number of records to be stored in memory as a second parameter when invoking the transformer. So on Linux, type in <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip 10000</pre> in order to set the threshold to 10,000 records. On a Windows machine, type in following: <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip 10000</pre> ebb6af4f519bb8a6a0405ec6d5c362d361b84ce7 609 608 2012-07-05T11:56:09Z JoergHoletschek 3 /* DarwinCore Archives */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Usage=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|775px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. ===Requirements=== Since this tool is currently a standalone application, it doesn't need to be run on the same machine as the BioCASe Provider Software. But of course it can - if you're running BioCASe on a publicly available server and plan to use DwCAs for production, the BioCASe server is definitely the right place. * Java Runtime Environment 1.5 or later installed * 2+G of RAM available * 100M free disk space * One or several ABCD2.06 Archives (created with BPS3+) ===Installation=== # If you don't have a Java Runtime Environment (version 1.5 or later) installed on your machine, get one [http://www.java.com/de/download/ here]. # Download the DwCA Creator from the [http://www.biocase.org/products/provider_software/index.shtml#DwCA_Creator BioCASe Website] and unzip the archive file to your machine. This will create a folder ''dwca'' for the tool. # Test the installation by opening a terminal (''command prompt'' on Windows) and changing to the ''dwca'' folder. Start the tool by running the batch file ''dwca.bat'' on Windows or the shell script ''dwca.sh'' on Linux/MacOS. Since it doesn't do any processing, the test run should take only a few seconds and print some lines similar to following (you can ignore the warning ''No such file or directory'' in the last line): [[File:dwcaTest.png]] ===Usage=== The converter can be started with the scripts ''dwca.bat'' (on Windows) and ''dwca.sh'' (on Linux/MacOS) and usually takes one argument - the ABCD archive to be converted. The tool accepts both a file stored in the local file system and an archive available for download at a given URL. Let's assume you've created an ABCD2 dump for your collection. You've placed the archive file (named ''AlgaTerra_ABCD_2.06.zip'') in the same folder as the tool (''dwca''). To run the conversion on Linux, open a terminal window, change to the ''dwca'' folder and type in the following: <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command. Note that the base folder for the transformer is the ''kettle'' folder, so you need to specify ''..\'' to direct the transformer to its parent folder for finding the archive file (alternatively, you can put the archive file in the ''kettle'' subfolder and thus spare the ''..\''): <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip</pre> This will unzip the ABCD documents stored in the archive file, create the data files, EML document(s) and the archive descriptor(s): [[File:dwcaStartLocal.png]] Depending on the size of your collection (number of records), the amount of information published (number of concepts mapped) and the speed of your server (available memory, number of CPUs/cores), this will take from several seconds to some minutes. For large collections (millions of records), it can take up to hours. After each chunk of 50,000 records processed, a line indicating the progress will be printed, so you will be able to see if the tool is still running. However, independent of the amount of data to be processed, the last lines of the output will look similar to the following: [[File:dwcaEndLocal.png]] When creating an ABCD Archive with the BioCASe Provider Software, it becomes automatically available for download on the BioCASe installation after the process has finished. The DwCA Creator can download the archive directly from this location. So if you're BioCASe installation is accessible from the machine you're running the prototype on, you can spare the trouble of moving the file manually and provide the URL instead of a local file name. To get the download URL for an ABCD Archive you've created, find the archive in question in the list of existing archives in the configuration tool (see [[Archiving#XML_Archives|here]] for more on that). Right-click on the Download link and choose ''Copy link''. Then open a terminal window for the creator tool and provide the link intead of a file name: <pre>./dwca.sh http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command: <pre>dwca.bat http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> ===Output=== The DwCA(s) created can be found in the folder ''output'' created in the base directory (''dwca''). Remember that there will be one DwCA per dataset. So depending upon how many datasets are stored in the source ABCD archive, there will be one or several files in the output folder. On Linux and MacOS, the shell script ''dwca.sh'' will zip all files that make up one DarwinCore Archive into an archive file named after the dataset title. So you could send this files directly to a potential DwCA consumer: [[File:dwcaOutputFolderLinux.png]] On Windows, the batch file ''dwca.bat'' of the prototype doesn't do that zipping. So if you go to the output file after the conversion has finished, you will see several text files and XML documents, all named after the pattern ''<dataset> <filename>''. So for the example above, the output folder would look like this: [[File:dwcaOutputFolderWindows.png]] In order to create a DwCA that can be sent to a consumer, please do the follwing for each ''<dataset>'': # Note down the dataset title, # Remove the dataset title in the file names (''<dataset> <filename>'' becomes ''<filename>''), # Create a zip named ''<dataset>.zip'' with all the files you just renamed. This lack of functionality is due to the limited capabilities of Windows batch files. Once integrated into the Provider Software, you won't need to zip up files manually. ===Trouble Shooting=== If you run into problems when using this tool, please [[Special:Contact|contact us]] for help. The default settings of the transormation should be working for small and medium-sized collections (up to 500,000 records). If the processing is cancelled with an OutOfMemoryError, you should try the follwing: '''Increasing the maximum memory used by the transformer'''<br> Per default, the transformer will use up to 1GB of RAM for the transformation process. In case of an OutOfMemoryError, you should increase this limit to 2GB. Don't be stingy with memory, since it will be only used during the tranformation process and be freed afterwards. Moreover, this is only the upper limit, so actual memory usage might well stay below this limit. To allow the transformer to use more memory, go to the folder ''kettle'' of your installation. Open the file ''kitchen.sh'' (Linux/MacOS) or ''kitchen.bat'' (Windows) in a text editor. Use the ''Find'' feature of the editor to go to the spot in file that specifies the memory limit. Replace the value ''1024'' with the amount of memory you want to allow the transformation to use, so e.g. ''2048'' for 2GB: [[File:dwcaSettingMemory.png]] Save the file and restart the transformation. '''Lowering the number of records stored in memory'''<br> Per default, Kettle will store up to 100,000 records in memory before starting to use temporary files for sorting rows. You can set this threshold to a lower value to save some memory. However, be advised that this will result in heavy disk I/O on your machine, slowing down the transformation. Therefore you should first increase the memory limit before changing this setting. If you decide to do this, just add the desired maximim number of records to be stored in memory as a second parameter when invoking the transformer. So on Linux, type in <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip 10000</pre> in order to set the threshold to 10,000 records. On a Windows machine, type in following: <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip 10000</pre> f92e7e862553aa4aaab0ebeafe118a7ccd377632 610 609 2012-07-05T12:33:54Z JoergHoletschek 3 /* DarwinCore Archives */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Usage=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|775px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. ===Usage=== The converter can be started with the scripts ''dwca.bat'' (on Windows) and ''dwca.sh'' (on Linux/MacOS) and usually takes one argument - the ABCD archive to be converted. The tool accepts both a file stored in the local file system and an archive available for download at a given URL. Let's assume you've created an ABCD2 dump for your collection. You've placed the archive file (named ''AlgaTerra_ABCD_2.06.zip'') in the same folder as the tool (''dwca''). To run the conversion on Linux, open a terminal window, change to the ''dwca'' folder and type in the following: <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command. Note that the base folder for the transformer is the ''kettle'' folder, so you need to specify ''..\'' to direct the transformer to its parent folder for finding the archive file (alternatively, you can put the archive file in the ''kettle'' subfolder and thus spare the ''..\''): <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip</pre> This will unzip the ABCD documents stored in the archive file, create the data files, EML document(s) and the archive descriptor(s): [[File:dwcaStartLocal.png]] Depending on the size of your collection (number of records), the amount of information published (number of concepts mapped) and the speed of your server (available memory, number of CPUs/cores), this will take from several seconds to some minutes. For large collections (millions of records), it can take up to hours. After each chunk of 50,000 records processed, a line indicating the progress will be printed, so you will be able to see if the tool is still running. However, independent of the amount of data to be processed, the last lines of the output will look similar to the following: [[File:dwcaEndLocal.png]] When creating an ABCD Archive with the BioCASe Provider Software, it becomes automatically available for download on the BioCASe installation after the process has finished. The DwCA Creator can download the archive directly from this location. So if you're BioCASe installation is accessible from the machine you're running the prototype on, you can spare the trouble of moving the file manually and provide the URL instead of a local file name. To get the download URL for an ABCD Archive you've created, find the archive in question in the list of existing archives in the configuration tool (see [[Archiving#XML_Archives|here]] for more on that). Right-click on the Download link and choose ''Copy link''. Then open a terminal window for the creator tool and provide the link intead of a file name: <pre>./dwca.sh http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command: <pre>dwca.bat http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> ===Output=== The DwCA(s) created can be found in the folder ''output'' created in the base directory (''dwca''). Remember that there will be one DwCA per dataset. So depending upon how many datasets are stored in the source ABCD archive, there will be one or several files in the output folder. On Linux and MacOS, the shell script ''dwca.sh'' will zip all files that make up one DarwinCore Archive into an archive file named after the dataset title. So you could send this files directly to a potential DwCA consumer: [[File:dwcaOutputFolderLinux.png]] On Windows, the batch file ''dwca.bat'' of the prototype doesn't do that zipping. So if you go to the output file after the conversion has finished, you will see several text files and XML documents, all named after the pattern ''<dataset> <filename>''. So for the example above, the output folder would look like this: [[File:dwcaOutputFolderWindows.png]] In order to create a DwCA that can be sent to a consumer, please do the follwing for each ''<dataset>'': # Note down the dataset title, # Remove the dataset title in the file names (''<dataset> <filename>'' becomes ''<filename>''), # Create a zip named ''<dataset>.zip'' with all the files you just renamed. This lack of functionality is due to the limited capabilities of Windows batch files. Once integrated into the Provider Software, you won't need to zip up files manually. ===Trouble Shooting=== If you run into problems when using this tool, please [[Special:Contact|contact us]] for help. The default settings of the transormation should be working for small and medium-sized collections (up to 500,000 records). If the processing is cancelled with an OutOfMemoryError, you should try the follwing: '''Increasing the maximum memory used by the transformer'''<br> Per default, the transformer will use up to 1GB of RAM for the transformation process. In case of an OutOfMemoryError, you should increase this limit to 2GB. Don't be stingy with memory, since it will be only used during the tranformation process and be freed afterwards. Moreover, this is only the upper limit, so actual memory usage might well stay below this limit. To allow the transformer to use more memory, go to the folder ''kettle'' of your installation. Open the file ''kitchen.sh'' (Linux/MacOS) or ''kitchen.bat'' (Windows) in a text editor. Use the ''Find'' feature of the editor to go to the spot in file that specifies the memory limit. Replace the value ''1024'' with the amount of memory you want to allow the transformation to use, so e.g. ''2048'' for 2GB: [[File:dwcaSettingMemory.png]] Save the file and restart the transformation. '''Lowering the number of records stored in memory'''<br> Per default, Kettle will store up to 100,000 records in memory before starting to use temporary files for sorting rows. You can set this threshold to a lower value to save some memory. However, be advised that this will result in heavy disk I/O on your machine, slowing down the transformation. Therefore you should first increase the memory limit before changing this setting. If you decide to do this, just add the desired maximim number of records to be stored in memory as a second parameter when invoking the transformer. So on Linux, type in <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip 10000</pre> in order to set the threshold to 10,000 records. On a Windows machine, type in following: <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip 10000</pre> 4b683b3508a08c2ab49b588bdbb01551c7e75f6f 612 610 2012-07-05T12:36:21Z JoergHoletschek 3 /* Preparation */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Usage=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|775px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. ===Usage=== The converter can be started with the scripts ''dwca.bat'' (on Windows) and ''dwca.sh'' (on Linux/MacOS) and usually takes one argument - the ABCD archive to be converted. The tool accepts both a file stored in the local file system and an archive available for download at a given URL. Let's assume you've created an ABCD2 dump for your collection. You've placed the archive file (named ''AlgaTerra_ABCD_2.06.zip'') in the same folder as the tool (''dwca''). To run the conversion on Linux, open a terminal window, change to the ''dwca'' folder and type in the following: <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command. Note that the base folder for the transformer is the ''kettle'' folder, so you need to specify ''..\'' to direct the transformer to its parent folder for finding the archive file (alternatively, you can put the archive file in the ''kettle'' subfolder and thus spare the ''..\''): <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip</pre> This will unzip the ABCD documents stored in the archive file, create the data files, EML document(s) and the archive descriptor(s): [[File:dwcaStartLocal.png]] Depending on the size of your collection (number of records), the amount of information published (number of concepts mapped) and the speed of your server (available memory, number of CPUs/cores), this will take from several seconds to some minutes. For large collections (millions of records), it can take up to hours. After each chunk of 50,000 records processed, a line indicating the progress will be printed, so you will be able to see if the tool is still running. However, independent of the amount of data to be processed, the last lines of the output will look similar to the following: [[File:dwcaEndLocal.png]] When creating an ABCD Archive with the BioCASe Provider Software, it becomes automatically available for download on the BioCASe installation after the process has finished. The DwCA Creator can download the archive directly from this location. So if you're BioCASe installation is accessible from the machine you're running the prototype on, you can spare the trouble of moving the file manually and provide the URL instead of a local file name. To get the download URL for an ABCD Archive you've created, find the archive in question in the list of existing archives in the configuration tool (see [[Archiving#XML_Archives|here]] for more on that). Right-click on the Download link and choose ''Copy link''. Then open a terminal window for the creator tool and provide the link intead of a file name: <pre>./dwca.sh http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command: <pre>dwca.bat http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> ===Output=== The DwCA(s) created can be found in the folder ''output'' created in the base directory (''dwca''). Remember that there will be one DwCA per dataset. So depending upon how many datasets are stored in the source ABCD archive, there will be one or several files in the output folder. On Linux and MacOS, the shell script ''dwca.sh'' will zip all files that make up one DarwinCore Archive into an archive file named after the dataset title. So you could send this files directly to a potential DwCA consumer: [[File:dwcaOutputFolderLinux.png]] On Windows, the batch file ''dwca.bat'' of the prototype doesn't do that zipping. So if you go to the output file after the conversion has finished, you will see several text files and XML documents, all named after the pattern ''<dataset> <filename>''. So for the example above, the output folder would look like this: [[File:dwcaOutputFolderWindows.png]] In order to create a DwCA that can be sent to a consumer, please do the follwing for each ''<dataset>'': # Note down the dataset title, # Remove the dataset title in the file names (''<dataset> <filename>'' becomes ''<filename>''), # Create a zip named ''<dataset>.zip'' with all the files you just renamed. This lack of functionality is due to the limited capabilities of Windows batch files. Once integrated into the Provider Software, you won't need to zip up files manually. ===Trouble Shooting=== If you run into problems when using this tool, please [[Special:Contact|contact us]] for help. The default settings of the transormation should be working for small and medium-sized collections (up to 500,000 records). If the processing is cancelled with an OutOfMemoryError, you should try the follwing: '''Increasing the maximum memory used by the transformer'''<br> Per default, the transformer will use up to 1GB of RAM for the transformation process. In case of an OutOfMemoryError, you should increase this limit to 2GB. Don't be stingy with memory, since it will be only used during the tranformation process and be freed afterwards. Moreover, this is only the upper limit, so actual memory usage might well stay below this limit. To allow the transformer to use more memory, go to the folder ''kettle'' of your installation. Open the file ''kitchen.sh'' (Linux/MacOS) or ''kitchen.bat'' (Windows) in a text editor. Use the ''Find'' feature of the editor to go to the spot in file that specifies the memory limit. Replace the value ''1024'' with the amount of memory you want to allow the transformation to use, so e.g. ''2048'' for 2GB: [[File:dwcaSettingMemory.png]] Save the file and restart the transformation. '''Lowering the number of records stored in memory'''<br> Per default, Kettle will store up to 100,000 records in memory before starting to use temporary files for sorting rows. You can set this threshold to a lower value to save some memory. However, be advised that this will result in heavy disk I/O on your machine, slowing down the transformation. Therefore you should first increase the memory limit before changing this setting. If you decide to do this, just add the desired maximim number of records to be stored in memory as a second parameter when invoking the transformer. So on Linux, type in <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip 10000</pre> in order to set the threshold to 10,000 records. On a Windows machine, type in following: <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip 10000</pre> dc6b5d84cf63f7ac3988b3235d0f454b493d6185 613 612 2012-07-05T12:41:24Z JoergHoletschek 3 /* Preparation */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Usage=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|775px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png]] ===Usage=== The converter can be started with the scripts ''dwca.bat'' (on Windows) and ''dwca.sh'' (on Linux/MacOS) and usually takes one argument - the ABCD archive to be converted. The tool accepts both a file stored in the local file system and an archive available for download at a given URL. Let's assume you've created an ABCD2 dump for your collection. You've placed the archive file (named ''AlgaTerra_ABCD_2.06.zip'') in the same folder as the tool (''dwca''). To run the conversion on Linux, open a terminal window, change to the ''dwca'' folder and type in the following: <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command. Note that the base folder for the transformer is the ''kettle'' folder, so you need to specify ''..\'' to direct the transformer to its parent folder for finding the archive file (alternatively, you can put the archive file in the ''kettle'' subfolder and thus spare the ''..\''): <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip</pre> This will unzip the ABCD documents stored in the archive file, create the data files, EML document(s) and the archive descriptor(s): [[File:dwcaStartLocal.png]] Depending on the size of your collection (number of records), the amount of information published (number of concepts mapped) and the speed of your server (available memory, number of CPUs/cores), this will take from several seconds to some minutes. For large collections (millions of records), it can take up to hours. After each chunk of 50,000 records processed, a line indicating the progress will be printed, so you will be able to see if the tool is still running. However, independent of the amount of data to be processed, the last lines of the output will look similar to the following: [[File:dwcaEndLocal.png]] When creating an ABCD Archive with the BioCASe Provider Software, it becomes automatically available for download on the BioCASe installation after the process has finished. The DwCA Creator can download the archive directly from this location. So if you're BioCASe installation is accessible from the machine you're running the prototype on, you can spare the trouble of moving the file manually and provide the URL instead of a local file name. To get the download URL for an ABCD Archive you've created, find the archive in question in the list of existing archives in the configuration tool (see [[Archiving#XML_Archives|here]] for more on that). Right-click on the Download link and choose ''Copy link''. Then open a terminal window for the creator tool and provide the link intead of a file name: <pre>./dwca.sh http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command: <pre>dwca.bat http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> ===Output=== The DwCA(s) created can be found in the folder ''output'' created in the base directory (''dwca''). Remember that there will be one DwCA per dataset. So depending upon how many datasets are stored in the source ABCD archive, there will be one or several files in the output folder. On Linux and MacOS, the shell script ''dwca.sh'' will zip all files that make up one DarwinCore Archive into an archive file named after the dataset title. So you could send this files directly to a potential DwCA consumer: [[File:dwcaOutputFolderLinux.png]] On Windows, the batch file ''dwca.bat'' of the prototype doesn't do that zipping. So if you go to the output file after the conversion has finished, you will see several text files and XML documents, all named after the pattern ''<dataset> <filename>''. So for the example above, the output folder would look like this: [[File:dwcaOutputFolderWindows.png]] In order to create a DwCA that can be sent to a consumer, please do the follwing for each ''<dataset>'': # Note down the dataset title, # Remove the dataset title in the file names (''<dataset> <filename>'' becomes ''<filename>''), # Create a zip named ''<dataset>.zip'' with all the files you just renamed. This lack of functionality is due to the limited capabilities of Windows batch files. Once integrated into the Provider Software, you won't need to zip up files manually. ===Trouble Shooting=== If you run into problems when using this tool, please [[Special:Contact|contact us]] for help. The default settings of the transormation should be working for small and medium-sized collections (up to 500,000 records). If the processing is cancelled with an OutOfMemoryError, you should try the follwing: '''Increasing the maximum memory used by the transformer'''<br> Per default, the transformer will use up to 1GB of RAM for the transformation process. In case of an OutOfMemoryError, you should increase this limit to 2GB. Don't be stingy with memory, since it will be only used during the tranformation process and be freed afterwards. Moreover, this is only the upper limit, so actual memory usage might well stay below this limit. To allow the transformer to use more memory, go to the folder ''kettle'' of your installation. Open the file ''kitchen.sh'' (Linux/MacOS) or ''kitchen.bat'' (Windows) in a text editor. Use the ''Find'' feature of the editor to go to the spot in file that specifies the memory limit. Replace the value ''1024'' with the amount of memory you want to allow the transformation to use, so e.g. ''2048'' for 2GB: [[File:dwcaSettingMemory.png]] Save the file and restart the transformation. '''Lowering the number of records stored in memory'''<br> Per default, Kettle will store up to 100,000 records in memory before starting to use temporary files for sorting rows. You can set this threshold to a lower value to save some memory. However, be advised that this will result in heavy disk I/O on your machine, slowing down the transformation. Therefore you should first increase the memory limit before changing this setting. If you decide to do this, just add the desired maximim number of records to be stored in memory as a second parameter when invoking the transformer. So on Linux, type in <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip 10000</pre> in order to set the threshold to 10,000 records. On a Windows machine, type in following: <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip 10000</pre> 4cb1c3218236a156b1b99b5ef72195bbc15df578 616 613 2012-07-05T12:45:51Z JoergHoletschek 3 /* Preparation */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Usage=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|775px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] ===Usage=== The converter can be started with the scripts ''dwca.bat'' (on Windows) and ''dwca.sh'' (on Linux/MacOS) and usually takes one argument - the ABCD archive to be converted. The tool accepts both a file stored in the local file system and an archive available for download at a given URL. Let's assume you've created an ABCD2 dump for your collection. You've placed the archive file (named ''AlgaTerra_ABCD_2.06.zip'') in the same folder as the tool (''dwca''). To run the conversion on Linux, open a terminal window, change to the ''dwca'' folder and type in the following: <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command. Note that the base folder for the transformer is the ''kettle'' folder, so you need to specify ''..\'' to direct the transformer to its parent folder for finding the archive file (alternatively, you can put the archive file in the ''kettle'' subfolder and thus spare the ''..\''): <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip</pre> This will unzip the ABCD documents stored in the archive file, create the data files, EML document(s) and the archive descriptor(s): [[File:dwcaStartLocal.png]] Depending on the size of your collection (number of records), the amount of information published (number of concepts mapped) and the speed of your server (available memory, number of CPUs/cores), this will take from several seconds to some minutes. For large collections (millions of records), it can take up to hours. After each chunk of 50,000 records processed, a line indicating the progress will be printed, so you will be able to see if the tool is still running. However, independent of the amount of data to be processed, the last lines of the output will look similar to the following: [[File:dwcaEndLocal.png]] When creating an ABCD Archive with the BioCASe Provider Software, it becomes automatically available for download on the BioCASe installation after the process has finished. The DwCA Creator can download the archive directly from this location. So if you're BioCASe installation is accessible from the machine you're running the prototype on, you can spare the trouble of moving the file manually and provide the URL instead of a local file name. To get the download URL for an ABCD Archive you've created, find the archive in question in the list of existing archives in the configuration tool (see [[Archiving#XML_Archives|here]] for more on that). Right-click on the Download link and choose ''Copy link''. Then open a terminal window for the creator tool and provide the link intead of a file name: <pre>./dwca.sh http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command: <pre>dwca.bat http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> ===Output=== The DwCA(s) created can be found in the folder ''output'' created in the base directory (''dwca''). Remember that there will be one DwCA per dataset. So depending upon how many datasets are stored in the source ABCD archive, there will be one or several files in the output folder. On Linux and MacOS, the shell script ''dwca.sh'' will zip all files that make up one DarwinCore Archive into an archive file named after the dataset title. So you could send this files directly to a potential DwCA consumer: [[File:dwcaOutputFolderLinux.png]] On Windows, the batch file ''dwca.bat'' of the prototype doesn't do that zipping. So if you go to the output file after the conversion has finished, you will see several text files and XML documents, all named after the pattern ''<dataset> <filename>''. So for the example above, the output folder would look like this: [[File:dwcaOutputFolderWindows.png]] In order to create a DwCA that can be sent to a consumer, please do the follwing for each ''<dataset>'': # Note down the dataset title, # Remove the dataset title in the file names (''<dataset> <filename>'' becomes ''<filename>''), # Create a zip named ''<dataset>.zip'' with all the files you just renamed. This lack of functionality is due to the limited capabilities of Windows batch files. Once integrated into the Provider Software, you won't need to zip up files manually. ===Trouble Shooting=== If you run into problems when using this tool, please [[Special:Contact|contact us]] for help. The default settings of the transormation should be working for small and medium-sized collections (up to 500,000 records). If the processing is cancelled with an OutOfMemoryError, you should try the follwing: '''Increasing the maximum memory used by the transformer'''<br> Per default, the transformer will use up to 1GB of RAM for the transformation process. In case of an OutOfMemoryError, you should increase this limit to 2GB. Don't be stingy with memory, since it will be only used during the tranformation process and be freed afterwards. Moreover, this is only the upper limit, so actual memory usage might well stay below this limit. To allow the transformer to use more memory, go to the folder ''kettle'' of your installation. Open the file ''kitchen.sh'' (Linux/MacOS) or ''kitchen.bat'' (Windows) in a text editor. Use the ''Find'' feature of the editor to go to the spot in file that specifies the memory limit. Replace the value ''1024'' with the amount of memory you want to allow the transformation to use, so e.g. ''2048'' for 2GB: [[File:dwcaSettingMemory.png]] Save the file and restart the transformation. '''Lowering the number of records stored in memory'''<br> Per default, Kettle will store up to 100,000 records in memory before starting to use temporary files for sorting rows. You can set this threshold to a lower value to save some memory. However, be advised that this will result in heavy disk I/O on your machine, slowing down the transformation. Therefore you should first increase the memory limit before changing this setting. If you decide to do this, just add the desired maximim number of records to be stored in memory as a second parameter when invoking the transformer. So on Linux, type in <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip 10000</pre> in order to set the threshold to 10,000 records. On a Windows machine, type in following: <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip 10000</pre> 021800e997f1929e84b76b65e32f3ea4a14ac28d 617 616 2012-07-05T13:08:22Z JoergHoletschek 3 /* Preparation */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Usage=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|775px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Usage=== The converter can be started with the scripts ''dwca.bat'' (on Windows) and ''dwca.sh'' (on Linux/MacOS) and usually takes one argument - the ABCD archive to be converted. The tool accepts both a file stored in the local file system and an archive available for download at a given URL. Let's assume you've created an ABCD2 dump for your collection. You've placed the archive file (named ''AlgaTerra_ABCD_2.06.zip'') in the same folder as the tool (''dwca''). To run the conversion on Linux, open a terminal window, change to the ''dwca'' folder and type in the following: <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command. Note that the base folder for the transformer is the ''kettle'' folder, so you need to specify ''..\'' to direct the transformer to its parent folder for finding the archive file (alternatively, you can put the archive file in the ''kettle'' subfolder and thus spare the ''..\''): <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip</pre> This will unzip the ABCD documents stored in the archive file, create the data files, EML document(s) and the archive descriptor(s): [[File:dwcaStartLocal.png]] Depending on the size of your collection (number of records), the amount of information published (number of concepts mapped) and the speed of your server (available memory, number of CPUs/cores), this will take from several seconds to some minutes. For large collections (millions of records), it can take up to hours. After each chunk of 50,000 records processed, a line indicating the progress will be printed, so you will be able to see if the tool is still running. However, independent of the amount of data to be processed, the last lines of the output will look similar to the following: [[File:dwcaEndLocal.png]] When creating an ABCD Archive with the BioCASe Provider Software, it becomes automatically available for download on the BioCASe installation after the process has finished. The DwCA Creator can download the archive directly from this location. So if you're BioCASe installation is accessible from the machine you're running the prototype on, you can spare the trouble of moving the file manually and provide the URL instead of a local file name. To get the download URL for an ABCD Archive you've created, find the archive in question in the list of existing archives in the configuration tool (see [[Archiving#XML_Archives|here]] for more on that). Right-click on the Download link and choose ''Copy link''. Then open a terminal window for the creator tool and provide the link intead of a file name: <pre>./dwca.sh http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command: <pre>dwca.bat http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> ===Output=== The DwCA(s) created can be found in the folder ''output'' created in the base directory (''dwca''). Remember that there will be one DwCA per dataset. So depending upon how many datasets are stored in the source ABCD archive, there will be one or several files in the output folder. On Linux and MacOS, the shell script ''dwca.sh'' will zip all files that make up one DarwinCore Archive into an archive file named after the dataset title. So you could send this files directly to a potential DwCA consumer: [[File:dwcaOutputFolderLinux.png]] On Windows, the batch file ''dwca.bat'' of the prototype doesn't do that zipping. So if you go to the output file after the conversion has finished, you will see several text files and XML documents, all named after the pattern ''<dataset> <filename>''. So for the example above, the output folder would look like this: [[File:dwcaOutputFolderWindows.png]] In order to create a DwCA that can be sent to a consumer, please do the follwing for each ''<dataset>'': # Note down the dataset title, # Remove the dataset title in the file names (''<dataset> <filename>'' becomes ''<filename>''), # Create a zip named ''<dataset>.zip'' with all the files you just renamed. This lack of functionality is due to the limited capabilities of Windows batch files. Once integrated into the Provider Software, you won't need to zip up files manually. ===Trouble Shooting=== If you run into problems when using this tool, please [[Special:Contact|contact us]] for help. The default settings of the transormation should be working for small and medium-sized collections (up to 500,000 records). If the processing is cancelled with an OutOfMemoryError, you should try the follwing: '''Increasing the maximum memory used by the transformer'''<br> Per default, the transformer will use up to 1GB of RAM for the transformation process. In case of an OutOfMemoryError, you should increase this limit to 2GB. Don't be stingy with memory, since it will be only used during the tranformation process and be freed afterwards. Moreover, this is only the upper limit, so actual memory usage might well stay below this limit. To allow the transformer to use more memory, go to the folder ''kettle'' of your installation. Open the file ''kitchen.sh'' (Linux/MacOS) or ''kitchen.bat'' (Windows) in a text editor. Use the ''Find'' feature of the editor to go to the spot in file that specifies the memory limit. Replace the value ''1024'' with the amount of memory you want to allow the transformation to use, so e.g. ''2048'' for 2GB: [[File:dwcaSettingMemory.png]] Save the file and restart the transformation. '''Lowering the number of records stored in memory'''<br> Per default, Kettle will store up to 100,000 records in memory before starting to use temporary files for sorting rows. You can set this threshold to a lower value to save some memory. However, be advised that this will result in heavy disk I/O on your machine, slowing down the transformation. Therefore you should first increase the memory limit before changing this setting. If you decide to do this, just add the desired maximim number of records to be stored in memory as a second parameter when invoking the transformer. So on Linux, type in <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip 10000</pre> in order to set the threshold to 10,000 records. On a Windows machine, type in following: <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip 10000</pre> a897625c26c0521b31657b003181adadcba59de2 618 617 2012-07-05T13:09:53Z JoergHoletschek 3 /* Usage */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|775px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Usage=== The converter can be started with the scripts ''dwca.bat'' (on Windows) and ''dwca.sh'' (on Linux/MacOS) and usually takes one argument - the ABCD archive to be converted. The tool accepts both a file stored in the local file system and an archive available for download at a given URL. Let's assume you've created an ABCD2 dump for your collection. You've placed the archive file (named ''AlgaTerra_ABCD_2.06.zip'') in the same folder as the tool (''dwca''). To run the conversion on Linux, open a terminal window, change to the ''dwca'' folder and type in the following: <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command. Note that the base folder for the transformer is the ''kettle'' folder, so you need to specify ''..\'' to direct the transformer to its parent folder for finding the archive file (alternatively, you can put the archive file in the ''kettle'' subfolder and thus spare the ''..\''): <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip</pre> This will unzip the ABCD documents stored in the archive file, create the data files, EML document(s) and the archive descriptor(s): [[File:dwcaStartLocal.png]] Depending on the size of your collection (number of records), the amount of information published (number of concepts mapped) and the speed of your server (available memory, number of CPUs/cores), this will take from several seconds to some minutes. For large collections (millions of records), it can take up to hours. After each chunk of 50,000 records processed, a line indicating the progress will be printed, so you will be able to see if the tool is still running. However, independent of the amount of data to be processed, the last lines of the output will look similar to the following: [[File:dwcaEndLocal.png]] When creating an ABCD Archive with the BioCASe Provider Software, it becomes automatically available for download on the BioCASe installation after the process has finished. The DwCA Creator can download the archive directly from this location. So if you're BioCASe installation is accessible from the machine you're running the prototype on, you can spare the trouble of moving the file manually and provide the URL instead of a local file name. To get the download URL for an ABCD Archive you've created, find the archive in question in the list of existing archives in the configuration tool (see [[Archiving#XML_Archives|here]] for more on that). Right-click on the Download link and choose ''Copy link''. Then open a terminal window for the creator tool and provide the link intead of a file name: <pre>./dwca.sh http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command: <pre>dwca.bat http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> ===Output=== The DwCA(s) created can be found in the folder ''output'' created in the base directory (''dwca''). Remember that there will be one DwCA per dataset. So depending upon how many datasets are stored in the source ABCD archive, there will be one or several files in the output folder. On Linux and MacOS, the shell script ''dwca.sh'' will zip all files that make up one DarwinCore Archive into an archive file named after the dataset title. So you could send this files directly to a potential DwCA consumer: [[File:dwcaOutputFolderLinux.png]] On Windows, the batch file ''dwca.bat'' of the prototype doesn't do that zipping. So if you go to the output file after the conversion has finished, you will see several text files and XML documents, all named after the pattern ''<dataset> <filename>''. So for the example above, the output folder would look like this: [[File:dwcaOutputFolderWindows.png]] In order to create a DwCA that can be sent to a consumer, please do the follwing for each ''<dataset>'': # Note down the dataset title, # Remove the dataset title in the file names (''<dataset> <filename>'' becomes ''<filename>''), # Create a zip named ''<dataset>.zip'' with all the files you just renamed. This lack of functionality is due to the limited capabilities of Windows batch files. Once integrated into the Provider Software, you won't need to zip up files manually. ===Trouble Shooting=== If you run into problems when using this tool, please [[Special:Contact|contact us]] for help. The default settings of the transormation should be working for small and medium-sized collections (up to 500,000 records). If the processing is cancelled with an OutOfMemoryError, you should try the follwing: '''Increasing the maximum memory used by the transformer'''<br> Per default, the transformer will use up to 1GB of RAM for the transformation process. In case of an OutOfMemoryError, you should increase this limit to 2GB. Don't be stingy with memory, since it will be only used during the tranformation process and be freed afterwards. Moreover, this is only the upper limit, so actual memory usage might well stay below this limit. To allow the transformer to use more memory, go to the folder ''kettle'' of your installation. Open the file ''kitchen.sh'' (Linux/MacOS) or ''kitchen.bat'' (Windows) in a text editor. Use the ''Find'' feature of the editor to go to the spot in file that specifies the memory limit. Replace the value ''1024'' with the amount of memory you want to allow the transformation to use, so e.g. ''2048'' for 2GB: [[File:dwcaSettingMemory.png]] Save the file and restart the transformation. '''Lowering the number of records stored in memory'''<br> Per default, Kettle will store up to 100,000 records in memory before starting to use temporary files for sorting rows. You can set this threshold to a lower value to save some memory. However, be advised that this will result in heavy disk I/O on your machine, slowing down the transformation. Therefore you should first increase the memory limit before changing this setting. If you decide to do this, just add the desired maximim number of records to be stored in memory as a second parameter when invoking the transformer. So on Linux, type in <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip 10000</pre> in order to set the threshold to 10,000 records. On a Windows machine, type in following: <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip 10000</pre> 8105218f1e96575f036f355417b507a872ff24f7 619 618 2012-07-05T13:10:34Z JoergHoletschek 3 /* Usage */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|775px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== blubb ===Output=== The DwCA(s) created can be found in the folder ''output'' created in the base directory (''dwca''). Remember that there will be one DwCA per dataset. So depending upon how many datasets are stored in the source ABCD archive, there will be one or several files in the output folder. On Linux and MacOS, the shell script ''dwca.sh'' will zip all files that make up one DarwinCore Archive into an archive file named after the dataset title. So you could send this files directly to a potential DwCA consumer: [[File:dwcaOutputFolderLinux.png]] On Windows, the batch file ''dwca.bat'' of the prototype doesn't do that zipping. So if you go to the output file after the conversion has finished, you will see several text files and XML documents, all named after the pattern ''<dataset> <filename>''. So for the example above, the output folder would look like this: [[File:dwcaOutputFolderWindows.png]] In order to create a DwCA that can be sent to a consumer, please do the follwing for each ''<dataset>'': # Note down the dataset title, # Remove the dataset title in the file names (''<dataset> <filename>'' becomes ''<filename>''), # Create a zip named ''<dataset>.zip'' with all the files you just renamed. This lack of functionality is due to the limited capabilities of Windows batch files. Once integrated into the Provider Software, you won't need to zip up files manually. ===Trouble Shooting=== If you run into problems when using this tool, please [[Special:Contact|contact us]] for help. The default settings of the transormation should be working for small and medium-sized collections (up to 500,000 records). If the processing is cancelled with an OutOfMemoryError, you should try the follwing: '''Increasing the maximum memory used by the transformer'''<br> Per default, the transformer will use up to 1GB of RAM for the transformation process. In case of an OutOfMemoryError, you should increase this limit to 2GB. Don't be stingy with memory, since it will be only used during the tranformation process and be freed afterwards. Moreover, this is only the upper limit, so actual memory usage might well stay below this limit. To allow the transformer to use more memory, go to the folder ''kettle'' of your installation. Open the file ''kitchen.sh'' (Linux/MacOS) or ''kitchen.bat'' (Windows) in a text editor. Use the ''Find'' feature of the editor to go to the spot in file that specifies the memory limit. Replace the value ''1024'' with the amount of memory you want to allow the transformation to use, so e.g. ''2048'' for 2GB: [[File:dwcaSettingMemory.png]] Save the file and restart the transformation. '''Lowering the number of records stored in memory'''<br> Per default, Kettle will store up to 100,000 records in memory before starting to use temporary files for sorting rows. You can set this threshold to a lower value to save some memory. However, be advised that this will result in heavy disk I/O on your machine, slowing down the transformation. Therefore you should first increase the memory limit before changing this setting. If you decide to do this, just add the desired maximim number of records to be stored in memory as a second parameter when invoking the transformer. So on Linux, type in <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip 10000</pre> in order to set the threshold to 10,000 records. On a Windows machine, type in following: <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip 10000</pre> 94662d8f7bb6739d928d31dcf7dfb2967a4bab4c 0 100 595 2012-07-05T10:48:05Z JoergHoletschek 3 Created page with "With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of r..." wikitext text/x-wiki With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:archArchivingPage.png|border]] ===Usage=== In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create a new Archive</small> allows you to create a new archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:archProcess.png|border]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== XML Archives that use the ABCD 2.06 schema can be converted into DarwinCore Archives. Currently, this feature is only available as a standalone command-line tool. Once thoroughly tested, it will be intregrated into the BioCASe Provider Software and can be trigged from the configuration tool. ===Requirements=== Since this tool is currently a standalone application, it doesn't need to be run on the same machine as the BioCASe Provider Software. But of course it can - if you're running BioCASe on a publicly available server and plan to use DwCAs for production, the BioCASe server is definitely the right place. * Java Runtime Environment 1.5 or later installed * 2+G of RAM available * 100M free disk space * One or several ABCD2.06 Archives (created with BPS3+) ===Installation=== # If you don't have a Java Runtime Environment (version 1.5 or later) installed on your machine, get one [http://www.java.com/de/download/ here]. # Download the DwCA Creator from the [http://www.biocase.org/products/provider_software/index.shtml#DwCA_Creator BioCASe Website] and unzip the archive file to your machine. This will create a folder ''dwca'' for the tool. # Test the installation by opening a terminal (''command prompt'' on Windows) and changing to the ''dwca'' folder. Start the tool by running the batch file ''dwca.bat'' on Windows or the shell script ''dwca.sh'' on Linux/MacOS. Since it doesn't do any processing, the test run should take only a few seconds and print some lines similar to following (you can ignore the warning ''No such file or directory'' in the last line): [[File:dwcaTest.png]] ===Usage=== The converter can be started with the scripts ''dwca.bat'' (on Windows) and ''dwca.sh'' (on Linux/MacOS) and usually takes one argument - the ABCD archive to be converted. The tool accepts both a file stored in the local file system and an archive available for download at a given URL. Let's assume you've created an ABCD2 dump for your collection. You've placed the archive file (named ''AlgaTerra_ABCD_2.06.zip'') in the same folder as the tool (''dwca''). To run the conversion on Linux, open a terminal window, change to the ''dwca'' folder and type in the following: <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command. Note that the base folder for the transformer is the ''kettle'' folder, so you need to specify ''..\'' to direct the transformer to its parent folder for finding the archive file (alternatively, you can put the archive file in the ''kettle'' subfolder and thus spare the ''..\''): <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip</pre> This will unzip the ABCD documents stored in the archive file, create the data files, EML document(s) and the archive descriptor(s): [[File:dwcaStartLocal.png]] Depending on the size of your collection (number of records), the amount of information published (number of concepts mapped) and the speed of your server (available memory, number of CPUs/cores), this will take from several seconds to some minutes. For large collections (millions of records), it can take up to hours. After each chunk of 50,000 records processed, a line indicating the progress will be printed, so you will be able to see if the tool is still running. However, independent of the amount of data to be processed, the last lines of the output will look similar to the following: [[File:dwcaEndLocal.png]] When creating an ABCD Archive with the BioCASe Provider Software, it becomes automatically available for download on the BioCASe installation after the process has finished. The DwCA Creator can download the archive directly from this location. So if you're BioCASe installation is accessible from the machine you're running the prototype on, you can spare the trouble of moving the file manually and provide the URL instead of a local file name. To get the download URL for an ABCD Archive you've created, find the archive in question in the list of existing archives in the configuration tool (see [[Archiving#XML_Archives|here]] for more on that). Right-click on the Download link and choose ''Copy link''. Then open a terminal window for the creator tool and provide the link intead of a file name: <pre>./dwca.sh http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command: <pre>dwca.bat http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> ===Output=== The DwCA(s) created can be found in the folder ''output'' created in the base directory (''dwca''). Remember that there will be one DwCA per dataset. So depending upon how many datasets are stored in the source ABCD archive, there will be one or several files in the output folder. On Linux and MacOS, the shell script ''dwca.sh'' will zip all files that make up one DarwinCore Archive into an archive file named after the dataset title. So you could send this files directly to a potential DwCA consumer: [[File:dwcaOutputFolderLinux.png]] On Windows, the batch file ''dwca.bat'' of the prototype doesn't do that zipping. So if you go to the output file after the conversion has finished, you will see several text files and XML documents, all named after the pattern ''<dataset> <filename>''. So for the example above, the output folder would look like this: [[File:dwcaOutputFolderWindows.png]] In order to create a DwCA that can be sent to a consumer, please do the follwing for each ''<dataset>'': # Note down the dataset title, # Remove the dataset title in the file names (''<dataset> <filename>'' becomes ''<filename>''), # Create a zip named ''<dataset>.zip'' with all the files you just renamed. This lack of functionality is due to the limited capabilities of Windows batch files. Once integrated into the Provider Software, you won't need to zip up files manually. ===Trouble Shooting=== If you run into problems when using this tool, please [[Special:Contact|contact us]] for help. The default settings of the transormation should be working for small and medium-sized collections (up to 500,000 records). If the processing is cancelled with an OutOfMemoryError, you should try the follwing: '''Increasing the maximum memory used by the transformer'''<br> Per default, the transformer will use up to 1GB of RAM for the transformation process. In case of an OutOfMemoryError, you should increase this limit to 2GB. Don't be stingy with memory, since it will be only used during the tranformation process and be freed afterwards. Moreover, this is only the upper limit, so actual memory usage might well stay below this limit. To allow the transformer to use more memory, go to the folder ''kettle'' of your installation. Open the file ''kitchen.sh'' (Linux/MacOS) or ''kitchen.bat'' (Windows) in a text editor. Use the ''Find'' feature of the editor to go to the spot in file that specifies the memory limit. Replace the value ''1024'' with the amount of memory you want to allow the transformation to use, so e.g. ''2048'' for 2GB: [[File:dwcaSettingMemory.png]] Save the file and restart the transformation. '''Lowering the number of records stored in memory'''<br> Per default, Kettle will store up to 100,000 records in memory before starting to use temporary files for sorting rows. You can set this threshold to a lower value to save some memory. However, be advised that this will result in heavy disk I/O on your machine, slowing down the transformation. Therefore you should first increase the memory limit before changing this setting. If you decide to do this, just add the desired maximim number of records to be stored in memory as a second parameter when invoking the transformer. So on Linux, type in <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip 10000</pre> in order to set the threshold to 10,000 records. On a Windows machine, type in following: <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip 10000</pre> 73b512664de9db1b458c62df4bde1e5303ef7b36 6 101 598 2012-07-05T10:52:27Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 599 598 2012-07-05T10:56:12Z JoergHoletschek 3 uploaded a new version of &quot;[[File:XmlArchives.png]]&quot; wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 102 603 2012-07-05T11:06:49Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 103 611 2012-07-05T12:35:25Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 104 614 2012-07-05T12:41:40Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 615 614 2012-07-05T12:44:44Z JoergHoletschek 3 uploaded a new version of &quot;[[File:DwcaSystemSettings.png]]&quot; wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 0 80 620 619 2012-07-05T13:11:28Z JoergHoletschek 3 /* DarwinCore Archives */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|775px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== blubb 3101ecc0e232eec3e8a943da9a182eaab3f0abc3 621 620 2012-07-05T13:46:23Z JoergHoletschek 3 /* Creating an Archive */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|775px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. The XML archiving step and its parameters are described in detail in the xxx section above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border]] 164af7490fb7e30b5f164d55d1309014274f660b 622 621 2012-07-05T13:46:43Z JoergHoletschek 3 /* Creating an Archive */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. The XML archiving step and its parameters are described in detail in the xxx section above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border]] 00d62b9024821217cd5641fca7eac86a5d4529f5 624 622 2012-07-05T13:48:12Z JoergHoletschek 3 /* Creating an Archive */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. The XML archiving step and its parameters are described in detail in the xxx section above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] c83306d03a347ddf01290e8fdb8ee5f229e6e5e8 625 624 2012-07-05T13:51:37Z JoergHoletschek 3 /* Creating an Archive */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the xxx section above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] 2aad7e2fd0601adf0fde98778dd10bddbfc37754 626 625 2012-07-05T13:52:33Z JoergHoletschek 3 /* Creating an Archive */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] ae0c5327c45db080035f0593b025244a57d1979c 627 626 2012-07-05T14:00:23Z JoergHoletschek 3 /* Creating an Archive */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depeding on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. 801c0cfb602dab3bf7f7c84d35375834fa7f2ba5 628 627 2012-07-05T14:14:30Z JoergHoletschek 3 /* Creating an Archive */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depeding on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset names and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png]] ===Trouble Shooting=== When trying to find the error, always read the log from top to bottom. Once you've found and solved the first problem, don't pay atteions, then restart the transformation using the ''DwC'' link of the respective XML archive. fe1321f902f8d312dae9c6e957c8192b94449cdd 630 628 2012-07-05T14:17:16Z JoergHoletschek 3 /* Creating an Archive */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depeding on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== When trying to find the error, always read the log from top to bottom. Once you've found and solved the first problem, don't pay atteions, then restart the transformation using the ''DwC'' link of the respective XML archive. 6de828b5b048090fd4ea4a63ac7186e4640b5d79 631 630 2012-07-05T14:18:55Z JoergHoletschek 3 /* Creating an Archive */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== When trying to find the error, always read the log from top to bottom. Once you've found and solved the first problem, don't pay atteions, then restart the transformation using the ''DwC'' link of the respective XML archive. 69c28911476044f356ecb32a45225590a1021137 632 631 2012-07-05T14:31:49Z JoergHoletschek 3 /* Trouble Shooting */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: 334eb9ccf865e6dcc2aea8194fb071a4f084c3d5 633 632 2012-07-05T14:35:32Z JoergHoletschek 3 /* Trouble Shooting */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> 583ef6b072f75cc8f582e1ce8d9af5c5be88ae52 634 633 2012-07-05T14:53:33Z JoergHoletschek 3 /* Trouble Shooting */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. 14a659e3dcafd4ccac7e04b9ff2eaf5b765cfe1f 635 634 2012-07-05T14:59:04Z JoergHoletschek 3 /* Trouble Shooting */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. f72a5c2998445305a819e0c4af96c1b619801cf5 636 635 2012-07-05T15:08:30Z JoergHoletschek 3 /* Trouble Shooting */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. 4e5ffbe280bddbcce3bf6b8a1ceb9af98edd5a54 637 636 2012-07-05T15:59:04Z JoergHoletschek 3 /* Trouble Shooting */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. b9872dcc6c8ee7e7a528cc59ab46090cf31b0202 649 637 2012-10-02T15:41:55Z JoergHoletschek 3 wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == 2b4f2bbd09933da659085d5de70b94fcb3ea4ca4 650 649 2012-10-02T15:42:17Z JoergHoletschek 3 /* Automating Archiving */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.2.1 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. Usage and Parameters Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. These two parameters – dsa for indicating the data source name, pw for specifying the config tool password – are mandatory. If you call the script without dsa and pw specified, the script will exit with an error messages. Below you’ll find the list of parameters: Dsa Mandatory. Specifies the data source you want to administer. Pw Mandatory. Config tool password, either for the whole installation or the data source, if a data source specific password has been set. Action This specifies the task to be done. Can be one of the following values: Log This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. Cancel Cancels any running archiving process for the given data source. If no process is running, it will do nothing. XML Starts XML archiving. You can customize the process with these parameters: Schema (optional). Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the Schemas section on the overview page of the data source configuration. Sample values would be ABCD_2.06 and HISPID_5 (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace http://www.tdwg.org/schemas/abcd/2.06); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace http://www.chah.org.au/schemas/hispid/5); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. filesize. (Optional). Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. Errors (Optional). Number of subsequent errors that are allowed to occur during the archiving before the process is cancelled with an error. Default is 1. DWC: Starts XML archiving with subsequent DarwinCore Archive generations. You can use the same parameters as for XML archiving. Remember that Java needs to be installed for DarwinCore Archiving. Return values of the script The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: For the actions cancel, the script will return a message once the process has been cancelled successfully: Output for log: XML/DWC: If an archiving process has been started successfully (actions xml and dwc), it will not return any result and time out eventually. For these actions, use separate log requests to find out about the progress of the process. Examples Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be http://localhost/biocase/pywrapper.cgi?dsa=pontaurus&pw=foo for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. http://localhost/biocase/pywrapper.cgi?dsa=pontaurus&pw=foo&action=xml to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). http://localhost/biocase/pywrapper.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5 to start XML archiving for HIDPID 5 with filesize=500 http://localhost/biocase/pywrapper.cgi?dsa=pontaurus&pw=foo&action=xml to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. http://localhost/biocase/pywrapper.cgi?dsa=pontaurus&pw=foo&action=cancel to cancel any running XML or DwC archiving process. 67d55a893e79c12fd9f3e1217c37aba5a32aed07 651 650 2012-10-02T15:43:56Z JoergHoletschek 3 /* Automating Archiving */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.2.1 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. These two parameters – ''dsa'' for indicating the data source name, ''pw'' for specifying the config tool password – are mandatory. If you call the script without ''dsa'' and ''pw'' specified, the script will exit with an error messages. Below you’ll find the list of parameters: Dsa Mandatory. Specifies the data source you want to administer. Pw Mandatory. Config tool password, either for the whole installation or the data source, if a data source specific password has been set. Action This specifies the task to be done. Can be one of the following values: Log This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. Cancel Cancels any running archiving process for the given data source. If no process is running, it will do nothing. XML Starts XML archiving. You can customize the process with these parameters: Schema (optional). Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the Schemas section on the overview page of the data source configuration. Sample values would be ABCD_2.06 and HISPID_5 (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace http://www.tdwg.org/schemas/abcd/2.06); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace http://www.chah.org.au/schemas/hispid/5); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. filesize. (Optional). Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. Errors (Optional). Number of subsequent errors that are allowed to occur during the archiving before the process is cancelled with an error. Default is 1. DWC: Starts XML archiving with subsequent DarwinCore Archive generations. You can use the same parameters as for XML archiving. Remember that Java needs to be installed for DarwinCore Archiving. Return values of the script The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: For the actions cancel, the script will return a message once the process has been cancelled successfully: Output for log: XML/DWC: If an archiving process has been started successfully (actions xml and dwc), it will not return any result and time out eventually. For these actions, use separate log requests to find out about the progress of the process. Examples Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be http://localhost/biocase/pywrapper.cgi?dsa=pontaurus&pw=foo for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. http://localhost/biocase/pywrapper.cgi?dsa=pontaurus&pw=foo&action=xml to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). http://localhost/biocase/pywrapper.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5 to start XML archiving for HIDPID 5 with filesize=500 http://localhost/biocase/pywrapper.cgi?dsa=pontaurus&pw=foo&action=xml to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. http://localhost/biocase/pywrapper.cgi?dsa=pontaurus&pw=foo&action=cancel to cancel any running XML or DwC archiving process. 4f8c6cf37918a0f8d9c349e744ecd404b8e28a03 652 651 2012-10-02T16:24:36Z JoergHoletschek 3 /* Automating Archiving */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.2.1 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'', ''filesize'' and ''errors''. |- | dwc |You can customize the process with the optional parameters ''schema'', ''filesize'' and ''errors''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. ;errors (Optional): Number of subsequent errors that are allowed to occur during the archiving before the process is cancelled with an error. Default is 1. Return values of the script The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: For the actions cancel, the script will return a message once the process has been cancelled successfully: Output for log: XML/DWC: If an archiving process has been started successfully (actions xml and dwc), it will not return any result and time out eventually. For these actions, use separate log requests to find out about the progress of the process. Examples Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be http://localhost/biocase/pywrapper.cgi?dsa=pontaurus&pw=foo for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. http://localhost/biocase/pywrapper.cgi?dsa=pontaurus&pw=foo&action=xml to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). http://localhost/biocase/pywrapper.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5 to start XML archiving for HIDPID 5 with filesize=500 http://localhost/biocase/pywrapper.cgi?dsa=pontaurus&pw=foo&action=xml to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. http://localhost/biocase/pywrapper.cgi?dsa=pontaurus&pw=foo&action=cancel to cancel any running XML or DwC archiving process. 805324d4f5c86e318a88271e736ce0d8ff1a17ef 653 652 2012-10-02T16:40:48Z JoergHoletschek 3 /* Automating Archiving */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.2.1 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'', ''filesize'' and ''errors''. |- | dwc |You can customize the process with the optional parameters ''schema'', ''filesize'' and ''errors''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. ;errors (Optional): Number of subsequent errors that are allowed to occur during the archiving before the process is cancelled with an error. Default is 1. ==Return values of the script== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate log requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' Examples Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be http://localhost/biocase/pywrapper.cgi?dsa=pontaurus&pw=foo for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. http://localhost/biocase/pywrapper.cgi?dsa=pontaurus&pw=foo&action=xml to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). http://localhost/biocase/pywrapper.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5 to start XML archiving for HIDPID 5 with filesize=500 http://localhost/biocase/pywrapper.cgi?dsa=pontaurus&pw=foo&action=xml to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. http://localhost/biocase/pywrapper.cgi?dsa=pontaurus&pw=foo&action=cancel to cancel any running XML or DwC archiving process. 3ff5432dbe3ce43a09a6b65d6e254f8034de6825 654 653 2012-10-02T16:41:57Z JoergHoletschek 3 wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.2.1 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'', ''filesize'' and ''errors''. |- | dwc |You can customize the process with the optional parameters ''schema'', ''filesize'' and ''errors''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. ;errors (Optional): Number of subsequent errors that are allowed to occur during the archiving before the process is cancelled with an error. Default is 1. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate log requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be http://localhost/biocase/pywrapper.cgi?dsa=pontaurus&pw=foo for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. http://localhost/biocase/pywrapper.cgi?dsa=pontaurus&pw=foo&action=xml to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). http://localhost/biocase/pywrapper.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5 to start XML archiving for HIDPID 5 with filesize=500 http://localhost/biocase/pywrapper.cgi?dsa=pontaurus&pw=foo&action=xml to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. http://localhost/biocase/pywrapper.cgi?dsa=pontaurus&pw=foo&action=cancel to cancel any running XML or DwC archiving process. 8755ae02bc817a0f718532491b3c302e3fac82a0 655 654 2012-10-02T16:43:40Z JoergHoletschek 3 /* Examples */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.2.1 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'', ''filesize'' and ''errors''. |- | dwc |You can customize the process with the optional parameters ''schema'', ''filesize'' and ''errors''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. ;errors (Optional): Number of subsequent errors that are allowed to occur during the archiving before the process is cancelled with an error. Default is 1. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate log requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/pywrapper.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/pywrapper.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/pywrapper.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/pywrapper.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/pywrapper.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. 52b09db32128eed8658b4670144f0d484a92f178 656 655 2012-10-02T16:44:00Z JoergHoletschek 3 /* Automating Archiving */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'', ''filesize'' and ''errors''. |- | dwc |You can customize the process with the optional parameters ''schema'', ''filesize'' and ''errors''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. ;errors (Optional): Number of subsequent errors that are allowed to occur during the archiving before the process is cancelled with an error. Default is 1. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate log requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/pywrapper.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/pywrapper.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/pywrapper.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/pywrapper.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/pywrapper.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. 53006791dfce06d58f5b1488ad7aaac43e74119f 657 656 2012-10-02T16:45:08Z JoergHoletschek 3 /* Examples */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'', ''filesize'' and ''errors''. |- | dwc |You can customize the process with the optional parameters ''schema'', ''filesize'' and ''errors''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. ;errors (Optional): Number of subsequent errors that are allowed to occur during the archiving before the process is cancelled with an error. Default is 1. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate log requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. f0de3b17fe7bc7a7957a7307d79e1f9048dc90e1 658 657 2012-10-02T16:47:58Z JoergHoletschek 3 /* Examples */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'', ''filesize'' and ''errors''. |- | dwc |You can customize the process with the optional parameters ''schema'', ''filesize'' and ''errors''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. ;errors (Optional): Number of subsequent errors that are allowed to occur during the archiving before the process is cancelled with an error. Default is 1. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate log requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. b39a5cf39ec63108f5d83c50a412c410dc575773 659 658 2012-10-02T16:50:37Z JoergHoletschek 3 /* Return values of the script */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'', ''filesize'' and ''errors''. |- | dwc |You can customize the process with the optional parameters ''schema'', ''filesize'' and ''errors''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. ;errors (Optional): Number of subsequent errors that are allowed to occur during the archiving before the process is cancelled with an error. Default is 1. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. be954b7251044c28161599ca7694857c2a0acc6b 660 659 2012-10-02T17:15:31Z JoergHoletschek 3 /* Automating Archiving */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.2.1 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'', ''filesize'' and ''errors''. |- | dwc |You can customize the process with the optional parameters ''schema'', ''filesize'' and ''errors''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. ;errors (Optional): Number of subsequent errors that are allowed to occur during the archiving before the process is cancelled with an error. Default is 1. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. 0a5675f7456ca7327ebf84e327679dccbace94d0 662 660 2012-10-04T15:29:48Z JoergHoletschek 3 /* Usage and Parameters */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.2.1 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'', ''filesize'' and ''errors''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'', ''filesize'' and ''errors''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. ;errors (Optional): Number of subsequent errors that are allowed to occur during the archiving before the process is cancelled with an error. Default is 1. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. 62618751756d4f4576862996215daa92b68797d7 663 662 2012-10-04T15:48:59Z JoergHoletschek 3 /* Automating Archiving */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'', ''filesize'' and ''errors''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'', ''filesize'' and ''errors''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. ;errors (Optional): Number of subsequent errors that are allowed to occur during the archiving before the process is cancelled with an error. Default is 1. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. b8191e78dbf3a6850d518241a94e92b8831a120c 6 105 623 2012-07-05T13:46:59Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 106 629 2012-07-05T14:15:17Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 0 100 638 595 2012-07-09T09:53:17Z JoergHoletschek 3 wikitext text/x-wiki '''This page describes the archiving features in old versions 3.0 and 3.1. If you're using version 3.2 or later, please read the [[Archiving|up-to-date version]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. [[File:archArchivingPage.png|border]] ===Usage=== In the box “Existing Archives” you will see a list of already existing XML archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create a new Archive</small> allows you to create a new archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:archProcess.png|border]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the respective entry listed under <small>in progress</small>. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. Always wait until the process has stopped before starting a new dump! As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Finished</small> list, if it’s still being processed, under <small>In progress</small>. Pressing the <small>Info</small> link next to an entry will display the log again. <small>Download</small> and <small>Remove</small> will do exactly what the links suggest. Even though you can create several archives in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. <strong><u>Important note:</u></strong> Be aware that a BioCASe web service can restrict the number of records per request! If that limit is below the paging size set for archiving, the paging size will be adjusted to the limit of the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== XML Archives that use the ABCD 2.06 schema can be converted into DarwinCore Archives. Currently, this feature is only available as a standalone command-line tool. Once thoroughly tested, it will be intregrated into the BioCASe Provider Software and can be trigged from the configuration tool. ===Requirements=== Since this tool is currently a standalone application, it doesn't need to be run on the same machine as the BioCASe Provider Software. But of course it can - if you're running BioCASe on a publicly available server and plan to use DwCAs for production, the BioCASe server is definitely the right place. * Java Runtime Environment 1.5 or later installed * 2+G of RAM available * 100M free disk space * One or several ABCD2.06 Archives (created with BPS3+) ===Installation=== # If you don't have a Java Runtime Environment (version 1.5 or later) installed on your machine, get one [http://www.java.com/de/download/ here]. # Download the DwCA Creator from the [http://www.biocase.org/products/provider_software/index.shtml#DwCA_Creator BioCASe Website] and unzip the archive file to your machine. This will create a folder ''dwca'' for the tool. # Test the installation by opening a terminal (''command prompt'' on Windows) and changing to the ''dwca'' folder. Start the tool by running the batch file ''dwca.bat'' on Windows or the shell script ''dwca.sh'' on Linux/MacOS. Since it doesn't do any processing, the test run should take only a few seconds and print some lines similar to following (you can ignore the warning ''No such file or directory'' in the last line): [[File:dwcaTest.png]] ===Usage=== The converter can be started with the scripts ''dwca.bat'' (on Windows) and ''dwca.sh'' (on Linux/MacOS) and usually takes one argument - the ABCD archive to be converted. The tool accepts both a file stored in the local file system and an archive available for download at a given URL. Let's assume you've created an ABCD2 dump for your collection. You've placed the archive file (named ''AlgaTerra_ABCD_2.06.zip'') in the same folder as the tool (''dwca''). To run the conversion on Linux, open a terminal window, change to the ''dwca'' folder and type in the following: <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command. Note that the base folder for the transformer is the ''kettle'' folder, so you need to specify ''..\'' to direct the transformer to its parent folder for finding the archive file (alternatively, you can put the archive file in the ''kettle'' subfolder and thus spare the ''..\''): <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip</pre> This will unzip the ABCD documents stored in the archive file, create the data files, EML document(s) and the archive descriptor(s): [[File:dwcaStartLocal.png]] Depending on the size of your collection (number of records), the amount of information published (number of concepts mapped) and the speed of your server (available memory, number of CPUs/cores), this will take from several seconds to some minutes. For large collections (millions of records), it can take up to hours. After each chunk of 50,000 records processed, a line indicating the progress will be printed, so you will be able to see if the tool is still running. However, independent of the amount of data to be processed, the last lines of the output will look similar to the following: [[File:dwcaEndLocal.png]] When creating an ABCD Archive with the BioCASe Provider Software, it becomes automatically available for download on the BioCASe installation after the process has finished. The DwCA Creator can download the archive directly from this location. So if you're BioCASe installation is accessible from the machine you're running the prototype on, you can spare the trouble of moving the file manually and provide the URL instead of a local file name. To get the download URL for an ABCD Archive you've created, find the archive in question in the list of existing archives in the configuration tool (see [[Archiving#XML_Archives|here]] for more on that). Right-click on the Download link and choose ''Copy link''. Then open a terminal window for the creator tool and provide the link intead of a file name: <pre>./dwca.sh http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> On a Windows machine, use the following command: <pre>dwca.bat http://ww3.bgbm.org/biocase/downloads/AlgaTerra/AlgaTerra_ABCD_2.06.zip</pre> ===Output=== The DwCA(s) created can be found in the folder ''output'' created in the base directory (''dwca''). Remember that there will be one DwCA per dataset. So depending upon how many datasets are stored in the source ABCD archive, there will be one or several files in the output folder. On Linux and MacOS, the shell script ''dwca.sh'' will zip all files that make up one DarwinCore Archive into an archive file named after the dataset title. So you could send this files directly to a potential DwCA consumer: [[File:dwcaOutputFolderLinux.png]] On Windows, the batch file ''dwca.bat'' of the prototype doesn't do that zipping. So if you go to the output file after the conversion has finished, you will see several text files and XML documents, all named after the pattern ''<dataset> <filename>''. So for the example above, the output folder would look like this: [[File:dwcaOutputFolderWindows.png]] In order to create a DwCA that can be sent to a consumer, please do the follwing for each ''<dataset>'': # Note down the dataset title, # Remove the dataset title in the file names (''<dataset> <filename>'' becomes ''<filename>''), # Create a zip named ''<dataset>.zip'' with all the files you just renamed. This lack of functionality is due to the limited capabilities of Windows batch files. Once integrated into the Provider Software, you won't need to zip up files manually. ===Trouble Shooting=== If you run into problems when using this tool, please [[Special:Contact|contact us]] for help. The default settings of the transormation should be working for small and medium-sized collections (up to 500,000 records). If the processing is cancelled with an OutOfMemoryError, you should try the follwing: '''Increasing the maximum memory used by the transformer'''<br> Per default, the transformer will use up to 1GB of RAM for the transformation process. In case of an OutOfMemoryError, you should increase this limit to 2GB. Don't be stingy with memory, since it will be only used during the tranformation process and be freed afterwards. Moreover, this is only the upper limit, so actual memory usage might well stay below this limit. To allow the transformer to use more memory, go to the folder ''kettle'' of your installation. Open the file ''kitchen.sh'' (Linux/MacOS) or ''kitchen.bat'' (Windows) in a text editor. Use the ''Find'' feature of the editor to go to the spot in file that specifies the memory limit. Replace the value ''1024'' with the amount of memory you want to allow the transformation to use, so e.g. ''2048'' for 2GB: [[File:dwcaSettingMemory.png]] Save the file and restart the transformation. '''Lowering the number of records stored in memory'''<br> Per default, Kettle will store up to 100,000 records in memory before starting to use temporary files for sorting rows. You can set this threshold to a lower value to save some memory. However, be advised that this will result in heavy disk I/O on your machine, slowing down the transformation. Therefore you should first increase the memory limit before changing this setting. If you decide to do this, just add the desired maximim number of records to be stored in memory as a second parameter when invoking the transformer. So on Linux, type in <pre>./dwca.sh AlgaTerra_ABCD_2.06.zip 10000</pre> in order to set the threshold to 10,000 records. On a Windows machine, type in following: <pre>dwca.bat ..\AlgaTerra_ABCD_2.06.zip 10000</pre> d3800d901146bb2cc2aadaed7df2eff9797af590 0 23 639 573 2012-07-09T11:33:22Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Version 3.2 [2012-07-09] === * DarwinCore Archives can now be generated directly from the config tool * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool b61ad7531ac9c55a3f4a337ac3d025f2834a8dc3 640 639 2012-07-09T11:34:27Z JoergHoletschek 3 /* Version 3.2 [2012-07-09] */ wikitext text/x-wiki == Current Version 3 == === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool a79a52ee88ee9b5a30368207a90d477ff64cab12 641 640 2012-07-24T13:42:50Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Bugfix for Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool e9752cda08206f245875d665bc088a499096afd6 642 641 2012-08-21T11:55:02Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Bugfixes ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool dc3b406ac6abefb6b606666bcf0ed54ce9b5e31f 643 642 2012-08-30T13:21:53Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Bugfixes ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool c44b94f9cedf4cbc32ffd698f99ff396e8c08798 645 643 2012-09-19T14:45:38Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 773fb442fd11ebb4a9f7b9431bd7535c056fa704 646 645 2012-09-20T16:06:25Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 1621ab6de50bb627729f9392e52aee9fe05cd773 647 646 2012-10-02T14:28:19Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Experimental: Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool d0aafee2e0397de3f809b36a0724c59917d789c0 648 647 2012-10-02T14:28:49Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Version 3.2.1 [Upcoming] === * Experimental: Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 9e0f6d874b05d0c4896dec633f37fa4b3e088797 664 648 2012-10-04T15:52:31Z JoergHoletschek 3 /* Version 3.2.1 [Upcoming] */ wikitext text/x-wiki == Current Version 3 == === Version 3.3 [2012-10-04] === * Experimental: Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 282382a63cd8cdcb35bc4af9319a15e9f1feb3a5 665 664 2012-10-04T16:23:20Z JoergHoletschek 3 /* Version 3.3 [2012-10-04] */ wikitext text/x-wiki == Current Version 3 == === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 8f5f09768e6f432ca99ed13d022cc0ec4b5fc2b7 666 665 2012-10-10T13:29:49Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Version 3.x.x [Upcoming] === * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 3c724685e6372d463d4487096c3b1f01ab08e153 667 666 2012-10-12T10:01:06Z JoergHoletschek 3 /* Version 3.x.x [Upcoming] */ wikitext text/x-wiki == Current Version 3 == === Version 3.x.x [Upcoming] === * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 89612cb3ed552dff007753c07da70776f504d5c1 668 667 2012-10-16T19:09:37Z JoergHoletschek 3 /* Version 3.x.x [Upcoming] */ wikitext text/x-wiki == Current Version 3 == === Version 3.x.x [Upcoming] === * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 2a261855b8a0f976c9ff10749cee3639c899c58d 669 668 2012-10-17T18:08:50Z JoergHoletschek 3 /* Version 3.x.x [Upcoming] */ wikitext text/x-wiki == Current Version 3 == === Version 3.x.x [Upcoming] === * Archiving: Archive name is now set automatically by BioCASe * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 55fde25a9693e82790ad5e2f8c64da16944712e8 0 52 644 584 2012-09-17T12:31:51Z JoergHoletschek 3 /* Installation */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I'm using Internet Information Server, and after creating the Virtual Directory for BioCASe, I get a “502.2 - Bad Gateway. The specified CGI application misbehaved by not returning a complete set of HTTP headers” error.=== This happens when you didn't correctly associate .cgi and .py files with Python. Make sure you've linked both files with Python and that the path contains the placeholders ''%s %s'' (see [[Installation#Internet_Information_Server|Linking IIS]]). ===I cannot find the installation package of the database connectivity module for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later of psycopg2. ===I'm trying to use BioCASe with Postgres; for this, I tried to install psycopg on Windows. But it doesn't work - either the install crashes, or it finishes, but the test libs page still reports psycopg2 as not installed. In the debug log I can find the error message "ImportError: DLL load failed (...)". What's wrong?=== We suspect this is caused by a bug in recent psycopg2 releases (find more [http://psycopg.lighthouseapp.com/projects/62710/tickets/20 here]). On our test installations, we managed to get round this issue by installing pycopg2 with administration rights. Do do this, right-click the installer file and choose "Run as administrator". ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== See [[Installation#Updating_an_existing_BioCASe_Installation|Updating an existing BioCASe Installation]]. ===I've updated my installation from a version prior to 2.6. Now, when I enter the DB structure, some of the table and column names are gone, the same happens in the mapping editor.=== Starting with version 2.6, BioCASe retrieves table and column names from the database for some database management systems (namely MySQL, SQL Server, Postgres, Foxpro, Access and Oracle). In older versions, you had to enter table/column names manually, and when you did the setup with such an early version, you probably didn't care for upper/lower cases. Since some DB management systems are case-sensitive, BioCASe can't ignore the differences and won't find these technically incorrect names in the lists retrieved from the database. The respective drop down boxes will therefore be empty. If that happens, just select the correct names from the drop down lists. ===When I try to create an archive, I get a "Permission denied" error. Why is that?=== For creating archives, you need to grant write permissions for the folders ''archive'' and ''www/downloads'' of your BioCASe installation to Python. You can check if permissions are set correctly in the test libs page (Start --> Utilities --> Test Libs). == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other only by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===I've mapped ABCD2. When testing the web service, I get a negative number for "RecordsDropped" in the response document. What does that mean?=== Same problem as above - it means you have an error in your mapping. Most probably you've mapped a non-repeatable ABCD concept to a table/column in your database that holds several entries per unit (occurrence) record. The BioCASe Provider Software will duplicate the respective unit records to be able to create valid ABCD documents - which is indicated by a negative number of "records dropped". Make sure the cardinality of your database relations match the structure (in respects of element repeatability) of ABCD to avoid this problem. ===I'm using pymssql to connect to a SQL Server database. Char(n) and Varchar(n) strings get truncated to 255 characters - what's wrong?=== This is known limitation of TDS protocol used by pymssql. Just change the data type for columns storing long string to TEXT or create a view that converts Char/Varchar columns to TEXT columns on the fly to work around this issue. ===I'm using pymssql to connect to a SQL Server database, BioCASe is running on a Linux distribution. When I do a Search or Scan request, I don't get any results (0 hits), even though there are definitely records in the database.=== That problem occurs on old versions of pymssql (1.0.2 or earlier) on Linux distributions. Update to version 2.0 or later to solve this. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. == Trouble Shooting == ===I’ve forgotten the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. 8850b958111c7af86b9f26cfe87631d4cb65b8bc 670 644 2012-11-08T12:31:14Z JoergHoletschek 3 /* Debugging */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I'm using Internet Information Server, and after creating the Virtual Directory for BioCASe, I get a “502.2 - Bad Gateway. The specified CGI application misbehaved by not returning a complete set of HTTP headers” error.=== This happens when you didn't correctly associate .cgi and .py files with Python. Make sure you've linked both files with Python and that the path contains the placeholders ''%s %s'' (see [[Installation#Internet_Information_Server|Linking IIS]]). ===I cannot find the installation package of the database connectivity module for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later of psycopg2. ===I'm trying to use BioCASe with Postgres; for this, I tried to install psycopg on Windows. But it doesn't work - either the install crashes, or it finishes, but the test libs page still reports psycopg2 as not installed. In the debug log I can find the error message "ImportError: DLL load failed (...)". What's wrong?=== We suspect this is caused by a bug in recent psycopg2 releases (find more [http://psycopg.lighthouseapp.com/projects/62710/tickets/20 here]). On our test installations, we managed to get round this issue by installing pycopg2 with administration rights. Do do this, right-click the installer file and choose "Run as administrator". ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== See [[Installation#Updating_an_existing_BioCASe_Installation|Updating an existing BioCASe Installation]]. ===I've updated my installation from a version prior to 2.6. Now, when I enter the DB structure, some of the table and column names are gone, the same happens in the mapping editor.=== Starting with version 2.6, BioCASe retrieves table and column names from the database for some database management systems (namely MySQL, SQL Server, Postgres, Foxpro, Access and Oracle). In older versions, you had to enter table/column names manually, and when you did the setup with such an early version, you probably didn't care for upper/lower cases. Since some DB management systems are case-sensitive, BioCASe can't ignore the differences and won't find these technically incorrect names in the lists retrieved from the database. The respective drop down boxes will therefore be empty. If that happens, just select the correct names from the drop down lists. ===When I try to create an archive, I get a "Permission denied" error. Why is that?=== For creating archives, you need to grant write permissions for the folders ''archive'' and ''www/downloads'' of your BioCASe installation to Python. You can check if permissions are set correctly in the test libs page (Start --> Utilities --> Test Libs). == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other only by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===I've mapped ABCD2. When testing the web service, I get a negative number for "RecordsDropped" in the response document. What does that mean?=== Same problem as above - it means you have an error in your mapping. Most probably you've mapped a non-repeatable ABCD concept to a table/column in your database that holds several entries per unit (occurrence) record. The BioCASe Provider Software will duplicate the respective unit records to be able to create valid ABCD documents - which is indicated by a negative number of "records dropped". Make sure the cardinality of your database relations match the structure (in respects of element repeatability) of ABCD to avoid this problem. ===I'm using pymssql to connect to a SQL Server database. Char(n) and Varchar(n) strings get truncated to 255 characters - what's wrong?=== This is known limitation of TDS protocol used by pymssql. Just change the data type for columns storing long string to TEXT or create a view that converts Char/Varchar columns to TEXT columns on the fly to work around this issue. ===I'm using pymssql to connect to a SQL Server database, BioCASe is running on a Linux distribution. When I do a Search or Scan request, I don't get any results (0 hits), even though there are definitely records in the database.=== That problem occurs on old versions of pymssql (1.0.2 or earlier) on Linux distributions. Update to version 2.0 or later to solve this. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===I am using BioCASe on a Windows Server. For some queries with the Local QueryForm, I get an error page that is obviously produced by the web server and not BioCASe. It reads "Error Code: 500 Internal Server Error. The request was rejected by the HTTP filter. Contact the server administrator. (12217)"=== This happens when BioCASe is run behind a Microsoft ISA Server/Microsoft Forefront Threat Management Gateway. To solve this problem, follow the instructions given in this article: http://support.microsoft.com/kb/837865. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. == Trouble Shooting == ===I’ve forgotten the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. 6839ea7a9e734afed1cd51bd9969409937b20064 0 23 671 669 2012-11-08T16:19:08Z JoergHoletschek 3 /* Version 3.x.x [Upcoming] */ wikitext text/x-wiki == Current Version 3 == === Version 3.x.x [Upcoming] === * Archiving: Archive name is now set automatically by BioCASe * An existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG and ABCD-DNA * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 5c734255b172b3fe3c2db2c8374420f392f19f28 672 671 2012-11-09T11:26:24Z JoergHoletschek 3 /* Version 3.x.x [Upcoming] */ wikitext text/x-wiki == Current Version 3 == === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 99466e42312ad42f011d4e74e7fa0f60fe2c7b17 673 672 2012-11-09T12:50:47Z JoergHoletschek 3 /* Version 3.3.1 [2012-11-09] */ wikitext text/x-wiki == Current Version 3 == === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool becc187767ab3998a6c6b8f29db6fd4b268c6204 674 673 2012-11-20T15:40:03Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * New dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. == Version 2 History == === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 69d15d4dcf7154ce3283a2e94cf34a21df6bb61f 675 674 2012-11-20T15:41:42Z JoergHoletschek 3 wikitext text/x-wiki == Current Version 3 == === Upcoming === * New dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool dbf25b4d2aa53c7def0e83887e106bb07d55a397 676 675 2012-11-23T14:43:12Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * New dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Settings page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool fa4957b8e9f3de53fa711d74036e9bb603b48f52 677 676 2012-11-26T09:07:08Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Settings page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool f7d71347d294674dc83655ce65b86b5f81d67487 678 677 2012-11-26T11:13:32Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Settings page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 6391a1883aae37c25ee1752c927634c583d2bb29 682 678 2012-12-04T16:30:34Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Corrections in layout of DB structure page in datasource configuration tool. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Settings page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 9026b30ae6228e72fded968d92aebe6ed0d4607d 683 682 2012-12-05T15:26:31Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Corrections in layout of DB structure page in datasource configuration tool. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Settings page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 6fc5dfbba1a19a1d7bec4d94aa0b57c06868e8c1 684 683 2012-12-05T15:32:33Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * XML archiving reimplemented: ** More compact code. ** Now directly calls BioCASe libraries instead of using the web service. Results in significantly improved archiving performance. ** XML documents are not pretty printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. * Corrections in layout of DB structure page in datasource configuration tool. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Settings page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 485355eb300133fe7a232b63095bdeba365d8cbd 685 684 2012-12-06T15:51:30Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * XML archiving reimplemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. * Corrections in layout of DB structure page in datasource configuration tool. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Settings page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 7d238ff4a7708ded9257d68f297de28ce5304557 686 685 2012-12-09T11:12:03Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * XML archiving reimplemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. * DwC archive transformation simplified. * Corrections in layout of DB structure page in datasource configuration tool. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Settings page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 96b62927e6d66a21b9e97f48baff64dd42d3a798 688 686 2012-12-18T11:01:26Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * XML archiving reimplemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. * DwC archive transformation simplified. * Corrections in layout of DB structure page in datasource configuration tool. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Settings page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 24feeef0ed01fc2e1eae0d3fc8080b77fd7260bf 689 688 2012-12-18T11:02:02Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Version 3.4 [to be released in Jan 2013]=== * XML archiving reimplemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. * DwC archive transformation simplified. * Corrections in layout of DB structure page in datasource configuration tool. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Settings page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool c23f5749ccfd2b175362586b32b8d0d26cf78a6d 690 689 2012-12-20T12:49:39Z JoergHoletschek 3 /* Version 3.4 [to be released in Jan 2013] */ wikitext text/x-wiki == Current Version 3 == === Version 3.4 [to be released in Jan 2013]=== * XML archiving reimplemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. * DwC archive transformation simplified. * Corrections in layout of DB structure page in datasource configuration tool. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up an empty, i.e. malformed response document). === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Settings page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool bdae2713b79f8185f8bbdf6718677ba620b2ecdd 691 690 2012-12-21T13:02:42Z JoergHoletschek 3 /* Version 3.4 [to be released in Jan 2013] */ wikitext text/x-wiki == Current Version 3 == === Version 3.4 [to be released in Jan 2013]=== * XML archiving reimplemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Corrections in layout of DB structure page in datasource configuration tool. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up an empty, i.e. malformed response document). === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Settings page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 3715daedb0b289ddfb3cfee06268136f928ac27d 692 691 2012-12-21T13:04:17Z JoergHoletschek 3 /* Version 3.4 [to be released in Jan 2013] */ wikitext text/x-wiki == Current Version 3 == === Version 3.4 [to be released in Jan 2013]=== * XML archiving reimplemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up an empty, i.e. malformed response document). === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Settings page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 9ac6c48cdbc8fb4cf85a69a4dd1b59c6359f858e 693 692 2012-12-21T13:32:32Z JoergHoletschek 3 /* Version 3.4 [to be released in Jan 2013] */ wikitext text/x-wiki == Current Version 3 == === Version 3.4 [to be released in Jan 2013]=== * XML archiving reimplemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Settings page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 9f8e0b2d9dfb99009bdbbbaf8d99860799a63d26 694 693 2012-12-21T13:48:52Z JoergHoletschek 3 /* Version 3.4 [to be released in Jan 2013] */ wikitext text/x-wiki == Current Version 3 == === Version 3.4 [to be released in Jan 2013]=== * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Settings page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 8d0941eb9669bb7b8a5cf5b5f68e9039019e1ae9 695 694 2012-12-21T17:53:51Z JoergHoletschek 3 /* Version 3.4 [to be released in Jan 2013] */ wikitext text/x-wiki == Current Version 3 == === Version 3.4 [to be released in Jan 2013]=== * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Settings page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 54286faf77d33acd4c2a2a7a5e76204c06198c0d 696 695 2013-01-22T17:59:26Z JoergHoletschek 3 /* Version 3.4 [to be released in Jan 2013] */ wikitext text/x-wiki == Current Version 3 == === Version 3.4 [to be released in Feb 2013]=== * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Settings page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 8d5a7bfad0554adc5af99cacef52606914c20ad1 697 696 2013-01-24T09:24:57Z JoergHoletschek 3 /* Version 3.4 [to be released in Feb 2013] */ wikitext text/x-wiki == Current Version 3 == === Version 3.4 [to be released in Feb 2013]=== * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Settings page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool a53904e6c5f6f2c12ccf4a68526135321edb5030 698 697 2013-01-24T09:27:31Z JoergHoletschek 3 /* Version 3.4 [to be released in Feb 2013] */ wikitext text/x-wiki == Current Version 3 == === Version 3.4 [to be released in Feb 2013]=== * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Settings page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 28f20bf03fec531d2fa8d715fa3663b8255588a7 701 698 2013-02-07T10:29:43Z JoergHoletschek 3 /* Version 3.4 [to be released in Feb 2013] */ wikitext text/x-wiki == Current Version 3 == === Version 3.4 [to be released in Feb 2013]=== * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Settings page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 3e37397f15c6e866a7e48d381c2b00c85939de36 702 701 2013-02-07T10:34:55Z JoergHoletschek 3 /* Version 3.4 [to be released in Feb 2013] */ wikitext text/x-wiki == Current Version 3 == === Version 3.4 [to be released in Feb 2013]=== * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Settings page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 888f0253d706c79ce696e79aebfdd204f1d840d5 703 702 2013-02-08T18:51:23Z JoergHoletschek 3 /* Version 3.4 [to be released in Feb 2013] */ wikitext text/x-wiki == Current Version 3 == === Version 3.4 [to be released in Feb 2013]=== * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Settings page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool a0cf8b7a5f6e9a4e5aa84f234071a5c2343fb555 704 703 2013-02-18T18:05:02Z JoergHoletschek 3 /* Version 3.4 [to be released in Feb 2013] */ wikitext text/x-wiki == Current Version 3 == === Version 3.4 [to be released in Feb 2013]=== * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Settings page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool bcaa8f87a4ef6e8a285ac68171de2e4fe026262b 705 704 2013-02-20T14:56:19Z JoergHoletschek 3 /* Version 3.4 [to be released in Feb 2013] */ wikitext text/x-wiki == Current Version 3 == === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Settings page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 3268ce15d5272bf9f6e96056c9176e87c97f8257 706 705 2013-02-28T15:07:09Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Fix in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Settings page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 1e421344369ce6cbed0481ee010a8f823fd022b0 707 706 2013-03-07T09:51:37Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Settings page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 2543795dfd6f1662cd3bcb7a543f2d13ce538000 708 707 2013-03-08T14:48:29Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Settings page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 61473faf0b3ecf200d5be97cb741bb1a8155b681 709 708 2013-03-11T10:28:30Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Settings page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool c7142b57be06a9447805f9304117036f9a7eba39 710 709 2013-03-18T12:18:32Z JoergHoletschek 3 /* Version 3.3.2 [2012-11-26] */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 8cd31c2d0eba011b2a98bd92325df0c126e92a27 711 710 2013-03-19T18:05:46Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 1f4d47f09d171f674fce4a225e35a2e0e15b4556 0 80 679 663 2012-12-03T16:28:04Z JoergHoletschek 3 /* Usage and Parameters */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'', ''filesize'' and ''errors''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'', ''filesize'' and ''errors''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. 75d707fbf7859e6dd5a925c04ed4459398a0174e 680 679 2012-12-03T16:29:20Z JoergHoletschek 3 /* Usage and Parameters */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. 3c9da73920e72a3dc20e8c684d75de2356a0fdbb 681 680 2012-12-03T16:29:39Z JoergHoletschek 3 /* Usage and Parameters */ wikitext text/x-wiki '''This page describes the archiving features in version 3.2 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. 5606ebcce5bbd9e10eed2ad97d19524bdba685da 712 681 2013-03-27T14:08:23Z JoergHoletschek 3 wikitext text/x-wiki '''This page describes the archiving features in versions 3.2 and 3.3. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for version 3.4 and later, go [[Archiving]].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. fa2904825b9fffe8af4de27b9095052641ec83ed 713 712 2013-03-27T14:18:46Z JoergHoletschek 3 wikitext text/x-wiki '''This page describes the archiving features in versions 3.2 and 3.3. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for version 3.4 and later, go [http://wiki.bgbm.org/bps/index.php/Archiving here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. d7d24c03196e18b111c76fa89d1514c389338faa 714 713 2013-03-27T14:21:56Z JoergHoletschek 3 wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in an archive, this file can be downloaded from your web server or sent to the harvester via email and ingested much faster. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, a separate XML Archive can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. Similar to a BioCASe web service, DwCAs can be used to publish data to the Global Biodiversity Information Facility (GBIF). A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as “ABCD/HISPID dumps”) can also be stored in one or several DarwinCore Archives. This is due to the fact that one BioCASe web service can publish several datasets, and a DarwinCore Archive can only store a single dataset. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD. Consequently, a DarwinCore Archive will store less information than an ABCD dump for a given web service. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. 6b12a21556a4185cc1f1aa6f0512b976aef43d38 715 714 2013-03-27T14:51:16Z JoergHoletschek 3 wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The Global Biodiversity Information Facility (GBIF), for example, can use this method for fast indexing of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchives.png|border|750px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. 28b00c2874b798c259cea4ed44cb496e880a8315 716 715 2013-03-27T15:01:18Z JoergHoletschek 3 wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The Global Biodiversity Information Facility (GBIF), for example, can use this method for fast indexing of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create an XML Archive for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchivesNew.png|border]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. 782b1297fcbdf32a853cc41039341186a6d1a2dd 718 716 2013-03-27T15:03:46Z JoergHoletschek 3 /* XML Archives */ wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The Global Biodiversity Information Facility (GBIF), for example, can use this method for fast indexing of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchivesNew.png|border|700px]] ===Creating an Archive=== In the box “Existing Archives” you will see a list of already existing XML and DwC archives. If you go to the archiving page for the first time, it will be empty. Once you’ve created archives, they will show up here and can be downloaded and removed by using the links next to the entries. <small>Create XML Archive</small> allows you to create a new XML archive for each schema mapped. Simply select the schema and press <small>Create Archive</small> (leave the defaults unless you have a good reason to change them). During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgress.png|border|825px]] You can always cancel the archiving process by pressing <small>Cancel</small> next to the name of the archive being currently processed. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>XML</small> list, or, if it’s still being processed, as <small>processing</small>. The links <small>Log</small>, <small>Dwnld</small> and <small>Rem</small> next to an existing archive will display the log, download or remove the file from the server. If the process failed for some reason (or you decided to cancel it), a new section <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create one archive at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. ffbe8bf15a22fe13da65f8353f467c88074bbb82 719 718 2013-03-27T15:18:20Z JoergHoletschek 3 /* Creating an Archive */ wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The Global Biodiversity Information Facility (GBIF), for example, can use this method for fast indexing of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchivesNew.png|border|700px]] ===Creating or Updating an Archive=== This sections allows you to create a new XML archive(s) for each schema mapped. Simply select the schema and press <small>Create XML Archive</small>. The paging size specifies how many records will be bundled into one XML document inside the archive; in most cases, the default 1000 will be OK. As mentioned above, archiving ABCD 2.06 or HISPID 5 is dataset-aware, resulting in one XML archive per dataset published. For all other schemas, a single archive file will be created. During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgressNew.png|border|825px]] You can always cancel the archiving process by pressing the <small>Cancel</small> link in the Status/Messages box. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Existing Archives</small> list, or, if it’s still being processed, as <small>Processing</small> in the Status/Messages box. If the process failed for some reason (or you decided to cancel it), a new box<small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create archives for one schema at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. ecd06ebed050daf5a2d13197e9221bc0f52f4a3e 720 719 2013-03-27T15:23:20Z JoergHoletschek 3 /* XML Archives */ wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The Global Biodiversity Information Facility (GBIF), for example, can use this method for fast indexing of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchivesNew.png|border|700px]] ===Creating or Updating an Archive=== This sections allows you to create new XML archive(s) for each schema mapped. Simply select the schema and press <small>Create XML Archive</small>. The paging size specifies how many records will be bundled into one XML document inside the archive; in most cases, the default 1000 will be OK. As mentioned above, archiving ABCD 2.06 or HISPID 5 is dataset-aware, resulting in one XML archive per dataset published. For all other schemas, a single archive file will be created. During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgressNew.png|border|825px]] You can always cancel the archiving process by pressing the <small>Cancel</small> link in the Status/Messages box. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Existing Archives</small> list, or, if it’s still being processed, as <small>Processing</small> in the Status/Messages box. If the process failed for some reason (or you decided to cancel it), a new box<small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create XML archives for one schema at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. 6e2ec1d102f1a2792c0daa24b7090ae250f6a0fd 0 52 687 670 2012-12-13T13:17:23Z JoergHoletschek 3 /* I'm using pymssql to connect to a SQL Server database, BioCASe is running on a Linux distribution. When I do a Search or Scan request, I don't get any results (0 hits), even though there are definitely records in the database. */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I'm using Internet Information Server, and after creating the Virtual Directory for BioCASe, I get a “502.2 - Bad Gateway. The specified CGI application misbehaved by not returning a complete set of HTTP headers” error.=== This happens when you didn't correctly associate .cgi and .py files with Python. Make sure you've linked both files with Python and that the path contains the placeholders ''%s %s'' (see [[Installation#Internet_Information_Server|Linking IIS]]). ===I cannot find the installation package of the database connectivity module for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later of psycopg2. ===I'm trying to use BioCASe with Postgres; for this, I tried to install psycopg on Windows. But it doesn't work - either the install crashes, or it finishes, but the test libs page still reports psycopg2 as not installed. In the debug log I can find the error message "ImportError: DLL load failed (...)". What's wrong?=== We suspect this is caused by a bug in recent psycopg2 releases (find more [http://psycopg.lighthouseapp.com/projects/62710/tickets/20 here]). On our test installations, we managed to get round this issue by installing pycopg2 with administration rights. Do do this, right-click the installer file and choose "Run as administrator". ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== See [[Installation#Updating_an_existing_BioCASe_Installation|Updating an existing BioCASe Installation]]. ===I've updated my installation from a version prior to 2.6. Now, when I enter the DB structure, some of the table and column names are gone, the same happens in the mapping editor.=== Starting with version 2.6, BioCASe retrieves table and column names from the database for some database management systems (namely MySQL, SQL Server, Postgres, Foxpro, Access and Oracle). In older versions, you had to enter table/column names manually, and when you did the setup with such an early version, you probably didn't care for upper/lower cases. Since some DB management systems are case-sensitive, BioCASe can't ignore the differences and won't find these technically incorrect names in the lists retrieved from the database. The respective drop down boxes will therefore be empty. If that happens, just select the correct names from the drop down lists. ===When I try to create an archive, I get a "Permission denied" error. Why is that?=== For creating archives, you need to grant write permissions for the folders ''archive'' and ''www/downloads'' of your BioCASe installation to Python. You can check if permissions are set correctly in the test libs page (Start --> Utilities --> Test Libs). == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other only by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===I've mapped ABCD2. When testing the web service, I get a negative number for "RecordsDropped" in the response document. What does that mean?=== Same problem as above - it means you have an error in your mapping. Most probably you've mapped a non-repeatable ABCD concept to a table/column in your database that holds several entries per unit (occurrence) record. The BioCASe Provider Software will duplicate the respective unit records to be able to create valid ABCD documents - which is indicated by a negative number of "records dropped". Make sure the cardinality of your database relations match the structure (in respects of element repeatability) of ABCD to avoid this problem. ===I'm using pymssql to connect to a SQL Server database. Char(n) and Varchar(n) strings get truncated to 255 characters - what's wrong?=== This is known limitation of TDS protocol used by pymssql. Just change the data type for columns storing long string to TEXT or create a view that converts Char/Varchar columns to TEXT columns on the fly to work around this issue. ===I'm using pymssql to connect to a SQL Server database, BioCASe is running on a Linux distribution. When I do a Search or Scan request, I don't get any results (0 hits), even though there are definitely records in the database.=== That problem occurs on old versions of pymssql (1.0.2 or earlier) on Linux distributions. Update to version 2.0 or later to solve this. If you still get empty result sets, this post might be helpful: http://code.google.com/p/pymssql/issues/detail?id=76. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===I am using BioCASe on a Windows Server. For some queries with the Local QueryForm, I get an error page that is obviously produced by the web server and not BioCASe. It reads "Error Code: 500 Internal Server Error. The request was rejected by the HTTP filter. Contact the server administrator. (12217)"=== This happens when BioCASe is run behind a Microsoft ISA Server/Microsoft Forefront Threat Management Gateway. To solve this problem, follow the instructions given in this article: http://support.microsoft.com/kb/837865. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. == Trouble Shooting == ===I’ve forgotten the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. 1281dc0a181d66b3e244131e296b0da52951592f 699 687 2013-01-24T09:33:06Z JoergHoletschek 3 /* I'm using pymssql to connect to a SQL Server database, BioCASe is running on a Linux distribution. When I do a Search or Scan request, I don't get any results (0 hits), even though there are definitely records in the database. */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I'm using Internet Information Server, and after creating the Virtual Directory for BioCASe, I get a “502.2 - Bad Gateway. The specified CGI application misbehaved by not returning a complete set of HTTP headers” error.=== This happens when you didn't correctly associate .cgi and .py files with Python. Make sure you've linked both files with Python and that the path contains the placeholders ''%s %s'' (see [[Installation#Internet_Information_Server|Linking IIS]]). ===I cannot find the installation package of the database connectivity module for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later of psycopg2. ===I'm trying to use BioCASe with Postgres; for this, I tried to install psycopg on Windows. But it doesn't work - either the install crashes, or it finishes, but the test libs page still reports psycopg2 as not installed. In the debug log I can find the error message "ImportError: DLL load failed (...)". What's wrong?=== We suspect this is caused by a bug in recent psycopg2 releases (find more [http://psycopg.lighthouseapp.com/projects/62710/tickets/20 here]). On our test installations, we managed to get round this issue by installing pycopg2 with administration rights. Do do this, right-click the installer file and choose "Run as administrator". ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== See [[Installation#Updating_an_existing_BioCASe_Installation|Updating an existing BioCASe Installation]]. ===I've updated my installation from a version prior to 2.6. Now, when I enter the DB structure, some of the table and column names are gone, the same happens in the mapping editor.=== Starting with version 2.6, BioCASe retrieves table and column names from the database for some database management systems (namely MySQL, SQL Server, Postgres, Foxpro, Access and Oracle). In older versions, you had to enter table/column names manually, and when you did the setup with such an early version, you probably didn't care for upper/lower cases. Since some DB management systems are case-sensitive, BioCASe can't ignore the differences and won't find these technically incorrect names in the lists retrieved from the database. The respective drop down boxes will therefore be empty. If that happens, just select the correct names from the drop down lists. ===When I try to create an archive, I get a "Permission denied" error. Why is that?=== For creating archives, you need to grant write permissions for the folders ''archive'' and ''www/downloads'' of your BioCASe installation to Python. You can check if permissions are set correctly in the test libs page (Start --> Utilities --> Test Libs). == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other only by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===I've mapped ABCD2. When testing the web service, I get a negative number for "RecordsDropped" in the response document. What does that mean?=== Same problem as above - it means you have an error in your mapping. Most probably you've mapped a non-repeatable ABCD concept to a table/column in your database that holds several entries per unit (occurrence) record. The BioCASe Provider Software will duplicate the respective unit records to be able to create valid ABCD documents - which is indicated by a negative number of "records dropped". Make sure the cardinality of your database relations match the structure (in respects of element repeatability) of ABCD to avoid this problem. ===I'm using pymssql to connect to a SQL Server database. Char(n) and Varchar(n) strings get truncated to 255 characters - what's wrong?=== This is known limitation of TDS protocol used by pymssql. Just change the data type for columns storing long string to TEXT or create a view that converts Char/Varchar columns to TEXT columns on the fly to work around this issue. ===I'm using pymssql to connect to a SQL Server database, BioCASe is running on a Linux distribution. When I do a Search or Scan request, I don't get any results (0 hits), even though there are definitely records in the database.=== That problem occurs on old versions of pymssql (1.0.2 or earlier) on Linux distributions. Update to version 2.0 or later to solve this. If you still get empty result sets, these posts might be helpful: * http://open-up.cybertaxonomy.africamuseum.be/forum_topic/important_if_you_use_biocase_bps_linux_platform_connect_ms_sql_server * http://code.google.com/p/pymssql/issues/detail?id=76. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===I am using BioCASe on a Windows Server. For some queries with the Local QueryForm, I get an error page that is obviously produced by the web server and not BioCASe. It reads "Error Code: 500 Internal Server Error. The request was rejected by the HTTP filter. Contact the server administrator. (12217)"=== This happens when BioCASe is run behind a Microsoft ISA Server/Microsoft Forefront Threat Management Gateway. To solve this problem, follow the instructions given in this article: http://support.microsoft.com/kb/837865. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. == Trouble Shooting == ===I’ve forgotten the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. 75103b6ba7d7deea07619e7e4115adf4d60f8b36 0 22 700 387 2013-01-31T11:12:27Z JoergHoletschek 3 /* Highly Recommended */ wikitext text/x-wiki This page lists the most commonly used ABCD 2.06 concepts. The ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a full list of ABCD concepts [http://wiki.tdwg.org/twiki/bin/view/ABCD/AbcdConcepts here]. == Mandatory == List of mandatory concepts. Without them being mapped the Provider Software will not be able to create valid ABCD document: <pre> DataSets └─ DataSet ├─ ContentContacts │ └─ ContentContact │ └─ Name ├─ TechnicalContacts │ └─ TechnicalContact │ └─ Name ├─ Metadata │ ├─ Description │ │ └─ Representation │ │ ├─ @language │ │ └─ Title │ └─ RevisionData │ └─ DateModified └─ Units └─ Unit ├─ SourceInstitutionID ├─ SourceID └─ UnitID </pre> == Highly Recommended == List of recommended ABCD concepts. If you can provide this data, please do so, they're commonly used entry points to your data. <pre> DataSets └─ DataSet ├─ ContentContacts │ └─ ContentContact │ ├─ Email │ └─ Address ├─ TechnicalContacts │ └─ TechnicalContact │ ├─ Email │ └─ Address ├─ Metadata │ ├─ Description │ │ └─ Representation │ │ └─ Details │ └─ Owners │ └─ Owner │ ├─ LogoURI │ ├─ URIs │ │ └─ URL │ ├─ Person │ │ └─ FullName │ └─ Organisation │ └─ Name │ └─ Representation │ └─ Text └─ Units └─ Unit ├─ Identifications │ └─ Identification │ └─ Result │ ├─ @PreferredFlag │ ├─ TaxonIdentified │ │ └─ ScientificName │ │ └─ FullScientificNameString │ └─ HigherTaxa │ └─ HigherTaxon │ ├─ HigherTaxonName │ └─ HigherTaxonRank ├─ RecordBasis └─ Gathering ├─ DateTime │ └─ ISODateTimeBegin ├─ Agents │ └─ GatheringAgent │ └─ Person │ └─ FullName ├─ LocalityText ├─ Country │ ├─ ISO3166Code │ └─ Name └─ SiteCoordinateSets └─ SiteCoordinates └─ CoordinatesLatLong ├─ LongitudeDecimal ├─ LatitudeDecimal └─ CoordinateErrorDistanceInMeters </pre> == Commonly Used == List of further commonly mapped ABCD concepts. <pre> DataSets └─ DataSet ├─ Metadata │ └─ IPRStatements │ ├─ TermsOfUseStatements │ │ └─ TermsOfUse │ │ └─ Text │ ├─ Copyrights │ │ └─ Copyright │ │ └─ Text │ ├─ IPRDeclarations │ │ └─ IPRDeclaration │ │ └─ Text │ ├─ Licenses │ │ └─ License │ │ └─ Text │ ├─ Disclaimers │ │ └─ Disclaimer │ │ └─ Text │ ├─ Citations │ │ └─ Citation │ │ └─ Text │ └─ Acknowledgements │ └─ Acknowledgement │ └─ Text └─ Units └─ Unit ├─ DateLastEdited ├─ Identifications │ └─ Identification │ ├─ Date │ │ └─ ISODateTimeBegin │ └─ Identifiers │ └─ Identifier │ └─ IdentifierPersonName │ └─ FullName ├─ KindOfUnit ├─ CollectorsFieldNumber ├─ Sex ├─ Age ├─ Notes ├─ RecordURI ├─ MultiMediaObjects │ └─ MultiMediaObject │ ├─ FileURI │ ├─ Comment │ └─ Context ├─ SpecimenUnit │ └─ NomenclaturalTypeDesignations │ └─ NomenclaturalTypeDesignation │ ├─ TypeStatus │ ├─ TypifiedName │ │ └─ FullScientificNameString │ └─ NomenclaturalReference │ ├─ TitleCitation │ └─ CitationDetail └─ Gathering ├─ Project │ └─ ProjectTitle ├─ Biotope │ └─ Name ├─ Altitude │ └─ MeasurementOrFactAtomised │ ├─ LowerValue │ └─ UnitOfMeasurement ├─ Depth │ └─ MeasurementOrFactAtomised │ ├─ LowerValue │ └─ UnitOfMeasurement └─ NamedAreas └─ NamedArea ├─ AreaName └─ AreaClass </pre> 48224faf9a463da1acabffe04606bdf6655de4b6 6 107 717 2013-03-27T15:01:39Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 108 721 2013-03-27T15:24:49Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 0 80 722 720 2013-03-27T15:25:25Z JoergHoletschek 3 /* Creating or Updating an Archive */ wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The Global Biodiversity Information Facility (GBIF), for example, can use this method for fast indexing of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchivesNew.png|border|700px]] ===Creating or Updating an Archive=== This sections allows you to create new XML archive(s) for each schema mapped. Simply select the schema and press <small>Create XML Archive</small>. The paging size specifies how many records will be bundled into one XML document inside the archive; in most cases, the default 1000 will be OK. As mentioned above, archiving ABCD 2.06 or HISPID 5 is dataset-aware, resulting in one XML archive per dataset published. For all other schemas, a single archive file will be created. During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgressNew.png|border|750px]] You can always cancel the archiving process by pressing the <small>Cancel</small> link in the Status/Messages box. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Existing Archives</small> list, or, if it’s still being processed, as <small>Processing</small> in the Status/Messages box. If the process failed for some reason (or you decided to cancel it), a new box<small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create XML archives for one schema at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ===Customizing the Archive=== There are four parameters for customizing the archiving process: <strong>Destination schema:</strong> Data schema that will be used for storing the published records in the archive. <strong>Paging size:</strong> Number of records stored in each document. The default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. <strong>Max number of consecutive errors:</strong> The default for this is “1”. That means that if an error occurs during data retrieval or XML/archive creation, the archiving process will be aborted. If you set this to a value larger than 1, the process will be cancelled after this number of errors. This allows you to complete the archiving in spite of errors, view the log file afterwards and have a look at the problematic records. Once you’ve corrected all errors breaking the dump, you should set the threshold to 1 and redo the archiving to create an archive storing all records. <strong>Archive file name:</strong> Name of the archive file to be created. The default will be constructed by concatenating web service name and data schema. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. db8d87c251f6100edd442ce9b7ae6e9bbc5d20cf 723 722 2013-03-27T15:29:17Z JoergHoletschek 3 /* XML Archives */ wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The Global Biodiversity Information Facility (GBIF), for example, can use this method for fast indexing of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchivesNew.png|border|700px]] ===Creating or Updating an Archive=== This sections allows you to create new XML archive(s) for each schema mapped. Simply select the schema and press <small>Create XML Archive</small>. As mentioned above, archiving ABCD 2.06 or HISPID 5 is dataset-aware, resulting in one XML archive per dataset published. For all other schemas, a single archive file will be created. The paging size specifies how many records will be bundled into one XML document inside the archive; the default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgressNew.png|border|750px]] You can always cancel the archiving process by pressing the <small>Cancel</small> link in the Status/Messages box. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Existing Archives</small> list, or, if it’s still being processed, as <small>Processing</small> in the Status/Messages box. If the process failed for some reason (or you decided to cancel it), a new box<small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create XML archives for one schema at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. b50dc13d603a433f5aaea95664d5e44c0303f158 724 723 2013-03-27T15:31:12Z JoergHoletschek 3 /* Creating or Updating an Archive */ wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The Global Biodiversity Information Facility (GBIF), for example, can use this method for fast indexing of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchivesNew.png|border|700px]] The section <small>Create/Update Archives</small> allows you to create new XML archive(s) for each schema mapped. Simply select the schema and press <small>Create XML Archive</small>. As mentioned above, archiving ABCD 2.06 or HISPID 5 is dataset-aware, resulting in one XML archive per dataset published. For all other schemas, a single archive file will be created. The paging size specifies how many records will be bundled into one XML document inside the archive; the default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgressNew.png|border|750px]] You can always cancel the archiving process by pressing the <small>Cancel</small> link in the Status/Messages box. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Existing Archives</small> list, or, if it’s still being processed, as <small>Processing</small> in the Status/Messages box. If the process failed for some reason (or you decided to cancel it), a new box<small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create XML archives for one schema at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. 799e52f4961594594c5179a59c8079c5548ce4b4 725 724 2013-03-27T15:37:41Z JoergHoletschek 3 /* XML Archives */ wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The Global Biodiversity Information Facility (GBIF), for example, can use this method for fast indexing of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchivesNew.png|border|700px]] The section <small>Create/Update Archives</small> allows you to create new XML archive(s) for each schema mapped. Simply select the schema and press <small>Create XML Archive</small>. As mentioned above, archiving ABCD 2.06 or HISPID 5 is dataset-aware, resulting in one XML archive per dataset published. For all other schemas, a single archive file will be created. The paging size specifies how many records will be bundled into one XML document inside the archive; the default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgressNew.png|border|750px]] You can always cancel the archiving process by pressing the <small>Cancel</small> link in the Status/Messages box. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Existing Archives</small> list, or, if it’s still being processed, as <small>Processing</small> in the Status/Messages box. If the process failed for some reason (or you decided to cancel it), a new box <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create XML archives for one schema at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. Since one BioCASe web service can publish several datasets and a DarwinCore Archive can only store a single dataset, DwC-archiving a web service might result in several DarwinCore Archives. DarwinCore archives are not created directly, but from the respective ABCD archive. So DwC-archiving a web service will usually first create an ACBD archive, which will then be converted into the respective DwC archive (alternatively, you can choose to convert an already existing ABCD archive into DwC). ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. b90174525db91e71b94288b00a29aad1dcd8484a 726 725 2013-03-27T15:42:58Z JoergHoletschek 3 /* DarwinCore Archives */ wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The Global Biodiversity Information Facility (GBIF), for example, can use this method for fast indexing of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchivesNew.png|border|700px]] The section <small>Create/Update Archives</small> allows you to create new XML archive(s) for each schema mapped. Simply select the schema and press <small>Create XML Archive</small>. As mentioned above, archiving ABCD 2.06 or HISPID 5 is dataset-aware, resulting in one XML archive per dataset published. For all other schemas, a single archive file will be created. The paging size specifies how many records will be bundled into one XML document inside the archive; the default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgressNew.png|border|750px]] You can always cancel the archiving process by pressing the <small>Cancel</small> link in the Status/Messages box. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Existing Archives</small> list, or, if it’s still being processed, as <small>Processing</small> in the Status/Messages box. If the process failed for some reason (or you decided to cancel it), a new box <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create XML archives for one schema at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. DarwinCore archives store only one dataset; so as with ABCD 2.06 and HISPID 5 XML archives, DwC-archiving a web service might result in several DarwinCore archives. DarwinCore archives are not created directly, but from the respective ABCD/HISPID archives. So DwC-archiving a web service will first create ACBD archives, which will be converted into the respective DwC archives subsequently. ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive manually, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the file name listed under ''Existing XML Archives''. The XML archiving step and its parameters are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link next to ''processing'' in the ''Existing archives'' box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from that ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. 172e2983608a867e0e176951b27c342c89310a4e 727 726 2013-03-27T15:50:09Z JoergHoletschek 3 /* Creating an Archive */ wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The Global Biodiversity Information Facility (GBIF), for example, can use this method for fast indexing of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchivesNew.png|border|700px]] The section <small>Create/Update Archives</small> allows you to create new XML archive(s) for each schema mapped. Simply select the schema and press <small>Create XML Archive</small>. As mentioned above, archiving ABCD 2.06 or HISPID 5 is dataset-aware, resulting in one XML archive per dataset published. For all other schemas, a single archive file will be created. The paging size specifies how many records will be bundled into one XML document inside the archive; the default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgressNew.png|border|750px]] You can always cancel the archiving process by pressing the <small>Cancel</small> link in the Status/Messages box. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Existing Archives</small> list, or, if it’s still being processed, as <small>Processing</small> in the Status/Messages box. If the process failed for some reason (or you decided to cancel it), a new box <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create XML archives for one schema at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. DarwinCore archives store only one dataset; so as with ABCD 2.06 and HISPID 5 XML archives, DwC-archiving a web service might result in several DarwinCore archives. DarwinCore archives are not created directly, but from the respective ABCD/HISPID archives. So DwC-archiving a web service will first create ACBD archives, which will be converted into the respective DwC archives subsequently. ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive before, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the archive name listed under ''Existing Archives''. The XML archiving step and the parameter <small>Paging size</small> are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgress.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link in the Status/Messages box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from the ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. Remember that several files will be generated if the web service publishes several datasets: [[File:dwcaArchiveList.png|555px]] ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. 279865130ed0cd2f0bc64ef060737ebe6fe15b84 728 727 2013-03-27T15:51:25Z JoergHoletschek 3 /* Creating an Archive */ wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The Global Biodiversity Information Facility (GBIF), for example, can use this method for fast indexing of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchivesNew.png|border|700px]] The section <small>Create/Update Archives</small> allows you to create new XML archive(s) for each schema mapped. Simply select the schema and press <small>Create XML Archive</small>. As mentioned above, archiving ABCD 2.06 or HISPID 5 is dataset-aware, resulting in one XML archive per dataset published. For all other schemas, a single archive file will be created. The paging size specifies how many records will be bundled into one XML document inside the archive; the default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgressNew.png|border|750px]] You can always cancel the archiving process by pressing the <small>Cancel</small> link in the Status/Messages box. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Existing Archives</small> list, or, if it’s still being processed, as <small>Processing</small> in the Status/Messages box. If the process failed for some reason (or you decided to cancel it), a new box <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create XML archives for one schema at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. DarwinCore archives store only one dataset; so as with ABCD 2.06 and HISPID 5 XML archives, DwC-archiving a web service might result in several DarwinCore archives. DarwinCore archives are not created directly, but from the respective ABCD/HISPID archives. So DwC-archiving a web service will first create ACBD archives, which will be converted into the respective DwC archives subsequently. ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive before, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the archive name listed under ''Existing Archives''. The XML archiving step and the parameter <small>Paging size</small> are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgressNew.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link in the Status/Messages box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from the ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: The DwC Archive creation completed successfully!'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. 80cdd211eaf2508e2ebefcfb13846c6d4d447ceb 729 728 2013-03-27T15:53:54Z JoergHoletschek 3 /* Creating an Archive */ wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The Global Biodiversity Information Facility (GBIF), for example, can use this method for fast indexing of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. [[File:xmlArchivesNew.png|border|700px]] The section <small>Create/Update Archives</small> allows you to create new XML archive(s) for each schema mapped. Simply select the schema and press <small>Create XML Archive</small>. As mentioned above, archiving ABCD 2.06 or HISPID 5 is dataset-aware, resulting in one XML archive per dataset published. For all other schemas, a single archive file will be created. The paging size specifies how many records will be bundled into one XML document inside the archive; the default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgressNew.png|border|750px]] You can always cancel the archiving process by pressing the <small>Cancel</small> link in the Status/Messages box. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Existing Archives</small> list, or, if it’s still being processed, as <small>Processing</small> in the Status/Messages box. If the process failed for some reason (or you decided to cancel it), a new box <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create XML archives for one schema at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. DarwinCore archives store only one dataset; so as with ABCD 2.06 and HISPID 5 XML archives, DwC-archiving a web service might result in several DarwinCore archives. DarwinCore archives are not created directly, but from the respective ABCD/HISPID archives. So DwC-archiving a web service will first create ACBD archives, which will be converted into the respective DwC archives subsequently. ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive before, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the archive name listed under ''Existing Archives''. The XML archiving step and the parameter <small>Paging size</small> are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgressNew.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link in the Status/Messages box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from the ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: Conversion finished.'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. 98a0826797037220669f264d3ddea55abfe44e19 731 729 2013-03-27T16:25:08Z JoergHoletschek 3 wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The Global Biodiversity Information Facility (GBIF), for example, can use this method for fast indexing of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==Managing Archives== The section <small>Existing Archives</small> allows you download, remove and convert archives or view the log of the archiving process: [[File:xmlArchivesNew.png|border|700px]] The archives are grouped by schema, with DarwinCore archives making up an additonal group. You can use the minus or plus symbol next to the group header to expand or collapse all lines of that group; for web services published dozens of datasets, this might be handy to reduce scrolling on the page. The <small>Log </small> link in the group header will open the log file for the respective archiving process in a separate tab. For each archive file, the number of records stored in this file, the date of creation and the file size are listed. XML archive may have a number printed in the <small>Dropped</small> column, meaning some records have been dropped due to mandatory reasons. This doesn't mean the archive is invalid, but you should review the mapping or your source data to find the reasons for these lost records. Clicking the archive name will download the archive file, so it can be saved to disk or opened with an archive viewing tool. Pressing the trash symbol at the very right will delete the archive; if you use the trash symbol in the group header, all archives of this schema/type will be removed. For ABCD 2.06 and HISPID 5 XML archives, you will see an additional <small>DwC</small> link next to the trash symbol. This allows you to selectively convert a single dataset from XML to DarwinCore archive. In contrast to using the <small>Create DwC Archive</small> button described below, this will spare the XML archiving of the whole web service and just create a DarwinCore archive for the selected dataset, which will be significantly faster. If you use the link in the group header, all archives of the schema will be converted (again without triggering XML archiving first). ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. The section <small>Create/Update Archives</small> allows you to create new XML archive(s) for each schema mapped. Simply select the schema and press <small>Create XML Archive</small>. As mentioned above, archiving ABCD 2.06 or HISPID 5 is dataset-aware, resulting in one XML archive per dataset published. For all other schemas, a single archive file will be created. The paging size specifies how many records will be bundled into one XML document inside the archive; the default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgressNew.png|border|750px]] You can always cancel the archiving process by pressing the <small>Cancel</small> link in the Status/Messages box. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Existing Archives</small> list, or, if it’s still being processed, as <small>Processing</small> in the Status/Messages box. If the process failed for some reason (or you decided to cancel it), a new box <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create XML archives for one schema at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. DarwinCore archives store only one dataset; so as with ABCD 2.06 and HISPID 5 XML archives, DwC-archiving a web service might result in several DarwinCore archives. DarwinCore archives are not created directly, but from the respective ABCD/HISPID archives. So DwC-archiving a web service will first create ACBD archives, which will be converted into the respective DwC archives subsequently. ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive before, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the archive name listed under ''Existing Archives''. The XML archiving step and the parameter <small>Paging size</small> are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgressNew.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link in the Status/Messages box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from the ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: Conversion finished.'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. 542032bf3e2e5e486e0f34ae3bb7920a7282642c 732 731 2013-03-27T16:26:09Z JoergHoletschek 3 /* Managing Archives */ wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The Global Biodiversity Information Facility (GBIF), for example, can use this method for fast indexing of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==Managing Archives== The section <small>Existing Archives</small> allows you download, remove and convert archives or view the log of the archiving process: [[File:xmlArchivesNew.png|border|700px]] The archives are grouped by schema, with DarwinCore archives making up an additonal group. You can use the minus or plus symbol next to the group header to expand or collapse all files of that group; for web services publishing dozens of datasets, this might be handy to reduce scrolling on the page. The <small>Log </small> link in the group header will open the log file for the respective archiving process in a separate tab. For each archive file, the number of records stored in this file, the date of creation and the file size are listed. XML archive may have a number printed in the <small>Dropped</small> column, meaning some records have been dropped due to mandatory reasons. This doesn't mean the archive is invalid, but you should review the mapping or your source data to find the reasons for these lost records. Clicking the archive name will download the archive file, so it can be saved to disk or opened with an archive viewing tool. Pressing the trash symbol at the very right will delete the archive; if you use the trash symbol in the group header, all archives of this schema/type will be removed. For ABCD 2.06 and HISPID 5 XML archives, you will see an additional <small>DwC</small> link next to the trash symbol. This allows you to selectively convert a single dataset from XML to DarwinCore archive. In contrast to using the <small>Create DwC Archive</small> button described below, this will spare the XML archiving of the whole web service and just create a DarwinCore archive for the selected dataset, which will be significantly faster. If you use the link in the group header, all archives of the schema will be converted (again without triggering XML archiving first). ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. The section <small>Create/Update Archives</small> allows you to create new XML archive(s) for each schema mapped. Simply select the schema and press <small>Create XML Archive</small>. As mentioned above, archiving ABCD 2.06 or HISPID 5 is dataset-aware, resulting in one XML archive per dataset published. For all other schemas, a single archive file will be created. The paging size specifies how many records will be bundled into one XML document inside the archive; the default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgressNew.png|border|750px]] You can always cancel the archiving process by pressing the <small>Cancel</small> link in the Status/Messages box. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Existing Archives</small> list, or, if it’s still being processed, as <small>Processing</small> in the Status/Messages box. If the process failed for some reason (or you decided to cancel it), a new box <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create XML archives for one schema at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. DarwinCore archives store only one dataset; so as with ABCD 2.06 and HISPID 5 XML archives, DwC-archiving a web service might result in several DarwinCore archives. DarwinCore archives are not created directly, but from the respective ABCD/HISPID archives. So DwC-archiving a web service will first create ACBD archives, which will be converted into the respective DwC archives subsequently. ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive before, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the archive name listed under ''Existing Archives''. The XML archiving step and the parameter <small>Paging size</small> are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgressNew.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link in the Status/Messages box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from the ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: Conversion finished.'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. d04340190eda9fec49dd6492f3e2e2d1da5d50cb 733 732 2013-03-27T16:27:01Z JoergHoletschek 3 /* Managing Archives */ wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The Global Biodiversity Information Facility (GBIF), for example, can use this method for fast indexing of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==Managing Archives== The section <small>Existing Archives</small> allows you download, remove and convert archives or view the log of the archiving process: [[File:xmlArchivesNew.png|border|700px]] The archives are grouped by schema, with DarwinCore archives making up an additonal group. You can use the minus or plus symbol next to the group header to expand or collapse all files of that group; for web services publishing dozens of datasets, this might be handy to reduce scrolling on the page. The <small>Log </small> link in the group header will open the log file for the respective archiving process in a separate tab. For each archive file, the number of records stored in this file, the date of creation and the file size are listed. XML archives may have a number printed in the <small>Dropped</small> column, meaning some records have been dropped due to mandatory reasons. This doesn't mean the archive is invalid, but you should review the mapping or your source data to find the reasons for these lost records. Clicking the archive name will download the archive file, so it can be saved to disk or opened with an archive viewing tool. Pressing the trash symbol at the very right will delete the archive; if you use the trash symbol in the group header, all archives of this schema/type will be removed. For ABCD 2.06 and HISPID 5 XML archives, you will see an additional <small>DwC</small> link next to the trash symbol. This allows you to selectively convert a single dataset from XML to DarwinCore archive. In contrast to using the <small>Create DwC Archive</small> button described below, this will spare the XML archiving of the whole web service and just create a DarwinCore archive for the selected dataset, which will be significantly faster. If you use the link in the group header, all archives of the schema will be converted (again without triggering XML archiving first). ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. The section <small>Create/Update Archives</small> allows you to create new XML archive(s) for each schema mapped. Simply select the schema and press <small>Create XML Archive</small>. As mentioned above, archiving ABCD 2.06 or HISPID 5 is dataset-aware, resulting in one XML archive per dataset published. For all other schemas, a single archive file will be created. The paging size specifies how many records will be bundled into one XML document inside the archive; the default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgressNew.png|border|750px]] You can always cancel the archiving process by pressing the <small>Cancel</small> link in the Status/Messages box. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Existing Archives</small> list, or, if it’s still being processed, as <small>Processing</small> in the Status/Messages box. If the process failed for some reason (or you decided to cancel it), a new box <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create XML archives for one schema at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. DarwinCore archives store only one dataset; so as with ABCD 2.06 and HISPID 5 XML archives, DwC-archiving a web service might result in several DarwinCore archives. DarwinCore archives are not created directly, but from the respective ABCD/HISPID archives. So DwC-archiving a web service will first create ACBD archives, which will be converted into the respective DwC archives subsequently. ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive before, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the archive name listed under ''Existing Archives''. The XML archiving step and the parameter <small>Paging size</small> are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgressNew.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link in the Status/Messages box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from the ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: Conversion finished.'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. 72c92adf32345ba4c69c2d74fe62c22d02da90be 734 733 2013-03-27T16:28:06Z JoergHoletschek 3 /* Managing Archives */ wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The Global Biodiversity Information Facility (GBIF), for example, can use this method for fast indexing of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==Managing Archives== The section <small>Existing Archives</small> allows you download, remove and convert archives or view the log of the archiving process: [[File:xmlArchivesNew.png|border|700px]] The archives are grouped by schema, with DarwinCore archives making up an additonal group. You can use the minus or plus symbol next to the group header to expand or collapse all files of that group; for web services publishing dozens of datasets, this might be handy to reduce scrolling on the page. The <small>Log </small> link in the group header will open the log file for the respective archiving process in a separate tab. For each archive file, the number of records stored in this file, the date of creation and the file size are listed. XML archives may have a number printed in the <small>Dropped</small> column, meaning some records have been dropped due to mandatory reasons. This doesn't mean the archive is invalid, but you should review the mapping or your source data to find the reasons for these lost records. Clicking the archive name will download the archive file, so it can be saved to disk or opened with an archive viewing tool. Pressing the trash symbol at the very right will delete the archive; if you use the trash symbol in the group header, all archives of this schema/type will be removed. For ABCD 2.06 and HISPID 5 XML archives, you will see an additional <small>DwC</small> link next to the trash symbol. This allows you to selectively convert a single dataset from XML to DarwinCore format. In contrast to using the <small>Create DwC Archive</small> button described below, this will spare the XML archiving of the whole web service and just create a DarwinCore archive for the selected dataset, which will be significantly faster. If you use the link in the group header, all archives of the schema will be converted (again without triggering XML archiving first). ==XML Archives== Once you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. The section <small>Create/Update Archives</small> allows you to create new XML archive(s) for each schema mapped. Simply select the schema and press <small>Create XML Archive</small>. As mentioned above, archiving ABCD 2.06 or HISPID 5 is dataset-aware, resulting in one XML archive per dataset published. For all other schemas, a single archive file will be created. The paging size specifies how many records will be bundled into one XML document inside the archive; the default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgressNew.png|border|750px]] You can always cancel the archiving process by pressing the <small>Cancel</small> link in the Status/Messages box. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Existing Archives</small> list, or, if it’s still being processed, as <small>Processing</small> in the Status/Messages box. If the process failed for some reason (or you decided to cancel it), a new box <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create XML archives for one schema at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. DarwinCore archives store only one dataset; so as with ABCD 2.06 and HISPID 5 XML archives, DwC-archiving a web service might result in several DarwinCore archives. DarwinCore archives are not created directly, but from the respective ABCD/HISPID archives. So DwC-archiving a web service will first create ACBD archives, which will be converted into the respective DwC archives subsequently. ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive before, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the archive name listed under ''Existing Archives''. The XML archiving step and the parameter <small>Paging size</small> are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgressNew.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link in the Status/Messages box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from the ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: Conversion finished.'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. 3847c77057b63cfe18a4f8f688ec6e524ba293d7 735 734 2013-03-27T16:29:10Z JoergHoletschek 3 /* XML Archives */ wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The Global Biodiversity Information Facility (GBIF), for example, can use this method for fast indexing of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==Managing Archives== The section <small>Existing Archives</small> allows you download, remove and convert archives or view the log of the archiving process: [[File:xmlArchivesNew.png|border|700px]] The archives are grouped by schema, with DarwinCore archives making up an additonal group. You can use the minus or plus symbol next to the group header to expand or collapse all files of that group; for web services publishing dozens of datasets, this might be handy to reduce scrolling on the page. The <small>Log </small> link in the group header will open the log file for the respective archiving process in a separate tab. For each archive file, the number of records stored in this file, the date of creation and the file size are listed. XML archives may have a number printed in the <small>Dropped</small> column, meaning some records have been dropped due to mandatory reasons. This doesn't mean the archive is invalid, but you should review the mapping or your source data to find the reasons for these lost records. Clicking the archive name will download the archive file, so it can be saved to disk or opened with an archive viewing tool. Pressing the trash symbol at the very right will delete the archive; if you use the trash symbol in the group header, all archives of this schema/type will be removed. For ABCD 2.06 and HISPID 5 XML archives, you will see an additional <small>DwC</small> link next to the trash symbol. This allows you to selectively convert a single dataset from XML to DarwinCore format. In contrast to using the <small>Create DwC Archive</small> button described below, this will spare the XML archiving of the whole web service and just create a DarwinCore archive for the selected dataset, which will be significantly faster. If you use the link in the group header, all archives of the schema will be converted (again without triggering XML archiving first). ==XML Archives== If you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. The section <small>Create/Update Archives</small> allows you to create new XML archive(s) for each schema mapped. Simply select the schema and press <small>Create XML Archive</small>. As mentioned above, archiving ABCD 2.06 or HISPID 5 is dataset-aware, resulting in one XML archive per dataset published. For all other schemas, a single archive file will be created. The paging size specifies how many records will be bundled into one XML document inside the archive; the default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgressNew.png|border|750px]] You can always cancel the archiving process by pressing the <small>Cancel</small> link in the Status/Messages box. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Existing Archives</small> list, or, if it’s still being processed, as <small>Processing</small> in the Status/Messages box. If the process failed for some reason (or you decided to cancel it), a new box <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create XML archives for one schema at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. DarwinCore archives store only one dataset; so as with ABCD 2.06 and HISPID 5 XML archives, DwC-archiving a web service might result in several DarwinCore archives. DarwinCore archives are not created directly, but from the respective ABCD/HISPID archives. So DwC-archiving a web service will first create ACBD archives, which will be converted into the respective DwC archives subsequently. ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive before, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the archive name listed under ''Existing Archives''. The XML archiving step and the parameter <small>Paging size</small> are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgressNew.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link in the Status/Messages box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from the ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: Conversion finished.'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. e1552323f07d4ea81d4e298ece4b13016af496c6 736 735 2013-03-27T16:32:45Z JoergHoletschek 3 /* Usage and Parameters */ wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later. If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The Global Biodiversity Information Facility (GBIF), for example, can use this method for fast indexing of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==Managing Archives== The section <small>Existing Archives</small> allows you download, remove and convert archives or view the log of the archiving process: [[File:xmlArchivesNew.png|border|700px]] The archives are grouped by schema, with DarwinCore archives making up an additonal group. You can use the minus or plus symbol next to the group header to expand or collapse all files of that group; for web services publishing dozens of datasets, this might be handy to reduce scrolling on the page. The <small>Log </small> link in the group header will open the log file for the respective archiving process in a separate tab. For each archive file, the number of records stored in this file, the date of creation and the file size are listed. XML archives may have a number printed in the <small>Dropped</small> column, meaning some records have been dropped due to mandatory reasons. This doesn't mean the archive is invalid, but you should review the mapping or your source data to find the reasons for these lost records. Clicking the archive name will download the archive file, so it can be saved to disk or opened with an archive viewing tool. Pressing the trash symbol at the very right will delete the archive; if you use the trash symbol in the group header, all archives of this schema/type will be removed. For ABCD 2.06 and HISPID 5 XML archives, you will see an additional <small>DwC</small> link next to the trash symbol. This allows you to selectively convert a single dataset from XML to DarwinCore format. In contrast to using the <small>Create DwC Archive</small> button described below, this will spare the XML archiving of the whole web service and just create a DarwinCore archive for the selected dataset, which will be significantly faster. If you use the link in the group header, all archives of the schema will be converted (again without triggering XML archiving first). ==XML Archives== If you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. The section <small>Create/Update Archives</small> allows you to create new XML archive(s) for each schema mapped. Simply select the schema and press <small>Create XML Archive</small>. As mentioned above, archiving ABCD 2.06 or HISPID 5 is dataset-aware, resulting in one XML archive per dataset published. For all other schemas, a single archive file will be created. The paging size specifies how many records will be bundled into one XML document inside the archive; the default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgressNew.png|border|750px]] You can always cancel the archiving process by pressing the <small>Cancel</small> link in the Status/Messages box. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Existing Archives</small> list, or, if it’s still being processed, as <small>Processing</small> in the Status/Messages box. If the process failed for some reason (or you decided to cancel it), a new box <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create XML archives for one schema at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. DarwinCore archives store only one dataset; so as with ABCD 2.06 and HISPID 5 XML archives, DwC-archiving a web service might result in several DarwinCore archives. DarwinCore archives are not created directly, but from the respective ABCD/HISPID archives. So DwC-archiving a web service will first create ACBD archives, which will be converted into the respective DwC archives subsequently. ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive before, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the archive name listed under ''Existing Archives''. The XML archiving step and the parameter <small>Paging size</small> are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgressNew.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link in the Status/Messages box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from the ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: Conversion finished.'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. This parameter corresponds to the paging size that can be set on the archiving page. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. 0144d341793f7a94581bf14929da3d1733b7b46b 737 736 2013-03-27T16:48:12Z JoergHoletschek 3 wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later.<br>If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The Global Biodiversity Information Facility (GBIF), for example, can use this method for fast indexing of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==Managing Archives== The section <small>Existing Archives</small> allows you download, remove and convert archives or view the log of the archiving process: [[File:xmlArchivesNew.png|border|700px]] The archives are grouped by schema, with DarwinCore archives making up an additonal group. You can use the minus or plus symbol next to the group header to expand or collapse all files of that group; for web services publishing dozens of datasets, this might be handy to reduce scrolling on the page. The <small>Log </small> link in the group header will open the log file for the respective archiving process in a separate tab. For each archive file, the number of records stored in this file, the date of creation and the file size are listed. XML archives may have a number printed in the <small>Dropped</small> column, meaning some records have been dropped due to mandatory reasons. This doesn't mean the archive is invalid, but you should review the mapping or your source data to find the reasons for these lost records. Clicking the archive name will download the archive file, so it can be saved to disk or opened with an archive viewing tool. Pressing the trash symbol at the very right will delete the archive; if you use the trash symbol in the group header, all archives of this schema/type will be removed. For ABCD 2.06 and HISPID 5 XML archives, you will see an additional <small>DwC</small> link next to the trash symbol. This allows you to selectively convert a single dataset from XML to DarwinCore format. In contrast to using the <small>Create DwC Archive</small> button described below, this will spare the XML archiving of the whole web service and just create a DarwinCore archive for the selected dataset, which will be significantly faster. If you use the link in the group header, all archives of the schema will be converted (again without triggering XML archiving first). ==XML Archives== If you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. The section <small>Create/Update Archives</small> allows you to create new XML archive(s) for each schema mapped. Simply select the schema and press <small>Create XML Archive</small>. As mentioned above, archiving ABCD 2.06 or HISPID 5 is dataset-aware, resulting in one XML archive per dataset published. For all other schemas, a single archive file will be created. The paging size specifies how many records will be bundled into one XML document inside the archive; the default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgressNew.png|border|750px]] You can always cancel the archiving process by pressing the <small>Cancel</small> link in the Status/Messages box. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Existing Archives</small> list, or, if it’s still being processed, as <small>Processing</small> in the Status/Messages box. If the process failed for some reason (or you decided to cancel it), a new box <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create XML archives for one schema at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. DarwinCore archives store only one dataset; so as with ABCD 2.06 and HISPID 5 XML archives, DwC-archiving a web service might result in several DarwinCore archives. DarwinCore archives are not created directly, but from the respective ABCD/HISPID archives. So DwC-archiving a web service will first create ACBD archives, which will be converted into the respective DwC archives subsequently. ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive before, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the archive name listed under ''Existing Archives''. The XML archiving step and the parameter <small>Paging size</small> are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgressNew.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link in the Status/Messages box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from the ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: Conversion finished.'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. This parameter corresponds to the paging size that can be set on the archiving page. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. ea639e26020c06522792b933631101e3cf3e9206 738 737 2013-03-27T16:50:09Z JoergHoletschek 3 /* Managing Archives */ wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later.<br>If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousands or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The Global Biodiversity Information Facility (GBIF), for example, can use this method for fast indexing of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==Managing Archives== The section <small>Existing Archives</small> allows you download, remove and convert archives or view the log of the archiving process: [[File:xmlArchivesNew.png|border|700px]] The archives are grouped by schema, with DarwinCore archives making up an additonal group. You can use the minus or plus symbol next to the group header to expand or collapse all files of that group; for web services publishing dozens of datasets, this might be handy to reduce scrolling on the page. The <small>Log </small> link in the group header will open the log file for the respective archiving process in a separate tab. For each archive file, the number of records stored in this file, the date of creation and the file size are listed. XML archives may have a number printed in the <small>Dropped</small> column, meaning some records have been dropped due to missing mandatory data elements. This doesn't mean the archive is invalid, but you should review the mapping or your source data to find the reasons for these lost records. Clicking the archive name will download the archive file, so it can be saved to disk or opened with an archive viewing tool. Pressing the trash symbol at the very right will delete the archive; if you use the trash symbol in the group header, all archives of this schema/type will be removed. For ABCD 2.06 and HISPID 5 XML archives, you will see an additional <small>DwC</small> link next to the trash symbol. This allows you to selectively convert a single dataset from XML to DarwinCore format. In contrast to using the <small>Create DwC Archive</small> button described below, this will spare the XML archiving of the whole web service and just create a DarwinCore archive for the selected dataset, which will be significantly faster. If you use the link in the group header, all archives of the schema will be converted (again without triggering XML archiving first). ==XML Archives== If you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. The section <small>Create/Update Archives</small> allows you to create new XML archive(s) for each schema mapped. Simply select the schema and press <small>Create XML Archive</small>. As mentioned above, archiving ABCD 2.06 or HISPID 5 is dataset-aware, resulting in one XML archive per dataset published. For all other schemas, a single archive file will be created. The paging size specifies how many records will be bundled into one XML document inside the archive; the default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgressNew.png|border|750px]] You can always cancel the archiving process by pressing the <small>Cancel</small> link in the Status/Messages box. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Existing Archives</small> list, or, if it’s still being processed, as <small>Processing</small> in the Status/Messages box. If the process failed for some reason (or you decided to cancel it), a new box <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create XML archives for one schema at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. DarwinCore archives store only one dataset; so as with ABCD 2.06 and HISPID 5 XML archives, DwC-archiving a web service might result in several DarwinCore archives. DarwinCore archives are not created directly, but from the respective ABCD/HISPID archives. So DwC-archiving a web service will first create ACBD archives, which will be converted into the respective DwC archives subsequently. ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the libary test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferrably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive before, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the archive name listed under ''Existing Archives''. The XML archiving step and the parameter <small>Paging size</small> are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgressNew.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link in the Status/Messages box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from the ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: Conversion finished.'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. This parameter corresponds to the paging size that can be set on the archiving page. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. b0b35c5b1870286ec44e0fabe78615a39d59343d 739 738 2013-03-27T17:10:43Z JoergHoletschek 3 wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later.<br>If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousand or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The Global Biodiversity Information Facility (GBIF), for example, can use this method for fast indexing of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==Managing Archives== The section <small>Existing Archives</small> allows you download, remove and convert archives or view the log of the archiving process: [[File:xmlArchivesNew.png|border|700px]] The archives are grouped by schema, with DarwinCore archives making up an additional group. You can use the minus or plus symbol next to the group header to expand or collapse all files of that group; for web services publishing dozens of datasets, this might be handy to reduce scrolling on the page. The <small>Log </small> link in the group header will open the log file for the respective archiving process in a separate tab. For each archive file, the number of records stored in this file, the date of creation and the file size are listed. XML archives may have a number printed in the <small>Dropped</small> column, meaning some records have been dropped due to missing mandatory data elements. This doesn't mean the archive is invalid, but you should review the mapping or your source data to find the reasons for these lost records. Clicking the archive name will download the archive file, so it can be saved to disk or opened with an archive viewing tool. Pressing the trash symbol at the very right will delete the archive; if you use the trash symbol in the group header, all archives of this schema/type will be removed. For ABCD 2.06 and HISPID 5 XML archives, you will see an additional <small>DwC</small> link next to the trash symbol. This allows you to selectively convert a single dataset from XML to DarwinCore format. In contrast to using the <small>Create DwC Archive</small> button described below, this will spare the XML archiving of the whole web service and just create a DarwinCore archive for the selected dataset, which will be significantly faster. If you use the link in the group header, all archives of the schema will be converted (again without triggering XML archiving first). ==XML Archives== If you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. The section <small>Create/Update Archives</small> allows you to create new XML archive(s) for each schema mapped. Simply select the schema and press <small>Create XML Archive</small>. As mentioned above, archiving ABCD 2.06 or HISPID 5 is dataset-aware, resulting in one XML archive per dataset published. For all other schemas, a single archive file will be created. The paging size specifies how many records will be bundled into one XML document inside the archive; the default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgressNew.png|border|750px]] You can always cancel the archiving process by pressing the <small>Cancel</small> link in the Status/Messages box. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Existing Archives</small> list, or, if it’s still being processed, as <small>Processing</small> in the Status/Messages box. If the process failed for some reason (or you decided to cancel it), a new box <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create XML archives for one schema at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. DarwinCore archives store only one dataset; so as with ABCD 2.06 and HISPID 5 XML archives, DwC-archiving a web service might result in several DarwinCore archives. DarwinCore archives are not created directly, but from the respective ABCD/HISPID archives. So DwC-archiving a web service will first create ACBD archives, which will be converted into the respective DwC archives subsequently. ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the library test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive before, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the archive name listed under ''Existing Archives''. The XML archiving step and the parameter <small>Paging size</small> are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgressNew.png|border|800px]] You can cancel the transformation process by clicking the <small>Cancel</small> link in the Status/Messages box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from the ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: Conversion finished.'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. This parameter corresponds to the paging size that can be set on the archiving page. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. c39582472ae439736f665b5e13a08f2d21bf9fa3 740 739 2013-03-27T17:26:32Z JoergHoletschek 3 /* Creating an Archive */ wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later.<br>If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousand or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The Global Biodiversity Information Facility (GBIF), for example, can use this method for fast indexing of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==Managing Archives== The section <small>Existing Archives</small> allows you download, remove and convert archives or view the log of the archiving process: [[File:xmlArchivesNew.png|border|700px]] The archives are grouped by schema, with DarwinCore archives making up an additional group. You can use the minus or plus symbol next to the group header to expand or collapse all files of that group; for web services publishing dozens of datasets, this might be handy to reduce scrolling on the page. The <small>Log </small> link in the group header will open the log file for the respective archiving process in a separate tab. For each archive file, the number of records stored in this file, the date of creation and the file size are listed. XML archives may have a number printed in the <small>Dropped</small> column, meaning some records have been dropped due to missing mandatory data elements. This doesn't mean the archive is invalid, but you should review the mapping or your source data to find the reasons for these lost records. Clicking the archive name will download the archive file, so it can be saved to disk or opened with an archive viewing tool. Pressing the trash symbol at the very right will delete the archive; if you use the trash symbol in the group header, all archives of this schema/type will be removed. For ABCD 2.06 and HISPID 5 XML archives, you will see an additional <small>DwC</small> link next to the trash symbol. This allows you to selectively convert a single dataset from XML to DarwinCore format. In contrast to using the <small>Create DwC Archive</small> button described below, this will spare the XML archiving of the whole web service and just create a DarwinCore archive for the selected dataset, which will be significantly faster. If you use the link in the group header, all archives of the schema will be converted (again without triggering XML archiving first). ==XML Archives== If you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. The section <small>Create/Update Archives</small> allows you to create new XML archive(s) for each schema mapped. Simply select the schema and press <small>Create XML Archive</small>. As mentioned above, archiving ABCD 2.06 or HISPID 5 is dataset-aware, resulting in one XML archive per dataset published. For all other schemas, a single archive file will be created. The paging size specifies how many records will be bundled into one XML document inside the archive; the default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgressNew.png|border|750px]] You can always cancel the archiving process by pressing the <small>Cancel</small> link in the Status/Messages box. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Existing Archives</small> list, or, if it’s still being processed, as <small>Processing</small> in the Status/Messages box. If the process failed for some reason (or you decided to cancel it), a new box <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create XML archives for one schema at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. DarwinCore archives store only one dataset; so as with ABCD 2.06 and HISPID 5 XML archives, DwC-archiving a web service might result in several DarwinCore archives. DarwinCore archives are not created directly, but from the respective ABCD/HISPID archives. So DwC-archiving a web service will first create ACBD archives, which will be converted into the respective DwC archives subsequently. ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the library test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive before, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the archive name listed under ''Existing Archives''. The XML archiving step and the parameter <small>Paging size</small> are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgressNew.png|border|850px]] You can cancel the transformation process by clicking the <small>Cancel</small> link in the Status/Messages box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from the ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: Conversion finished.'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. This parameter corresponds to the paging size that can be set on the archiving page. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. 35ed5150705c339ee0960e3291e040fcd235cb73 741 740 2013-03-27T17:26:57Z JoergHoletschek 3 /* Creating an Archive */ wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later.<br>If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousand or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The Global Biodiversity Information Facility (GBIF), for example, can use this method for fast indexing of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==Managing Archives== The section <small>Existing Archives</small> allows you download, remove and convert archives or view the log of the archiving process: [[File:xmlArchivesNew.png|border|700px]] The archives are grouped by schema, with DarwinCore archives making up an additional group. You can use the minus or plus symbol next to the group header to expand or collapse all files of that group; for web services publishing dozens of datasets, this might be handy to reduce scrolling on the page. The <small>Log </small> link in the group header will open the log file for the respective archiving process in a separate tab. For each archive file, the number of records stored in this file, the date of creation and the file size are listed. XML archives may have a number printed in the <small>Dropped</small> column, meaning some records have been dropped due to missing mandatory data elements. This doesn't mean the archive is invalid, but you should review the mapping or your source data to find the reasons for these lost records. Clicking the archive name will download the archive file, so it can be saved to disk or opened with an archive viewing tool. Pressing the trash symbol at the very right will delete the archive; if you use the trash symbol in the group header, all archives of this schema/type will be removed. For ABCD 2.06 and HISPID 5 XML archives, you will see an additional <small>DwC</small> link next to the trash symbol. This allows you to selectively convert a single dataset from XML to DarwinCore format. In contrast to using the <small>Create DwC Archive</small> button described below, this will spare the XML archiving of the whole web service and just create a DarwinCore archive for the selected dataset, which will be significantly faster. If you use the link in the group header, all archives of the schema will be converted (again without triggering XML archiving first). ==XML Archives== If you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. The section <small>Create/Update Archives</small> allows you to create new XML archive(s) for each schema mapped. Simply select the schema and press <small>Create XML Archive</small>. As mentioned above, archiving ABCD 2.06 or HISPID 5 is dataset-aware, resulting in one XML archive per dataset published. For all other schemas, a single archive file will be created. The paging size specifies how many records will be bundled into one XML document inside the archive; the default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgressNew.png|border|750px]] You can always cancel the archiving process by pressing the <small>Cancel</small> link in the Status/Messages box. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Existing Archives</small> list, or, if it’s still being processed, as <small>Processing</small> in the Status/Messages box. If the process failed for some reason (or you decided to cancel it), a new box <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create XML archives for one schema at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. DarwinCore archives store only one dataset; so as with ABCD 2.06 and HISPID 5 XML archives, DwC-archiving a web service might result in several DarwinCore archives. DarwinCore archives are not created directly, but from the respective ABCD/HISPID archives. So DwC-archiving a web service will first create ACBD archives, which will be converted into the respective DwC archives subsequently. ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the library test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive before, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the archive name listed under ''Existing Archives''. The XML archiving step and the parameter <small>Paging size</small> are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgressNew.png|border]] You can cancel the transformation process by clicking the <small>Cancel</small> link in the Status/Messages box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from the ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: Conversion finished.'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. This parameter corresponds to the paging size that can be set on the archiving page. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. 803ce403a6c1cf6bdabc216e58c6b3b62f6d483c 742 741 2013-03-27T17:27:27Z JoergHoletschek 3 /* XML Archives */ wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later.<br>If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousand or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The Global Biodiversity Information Facility (GBIF), for example, can use this method for fast indexing of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==Managing Archives== The section <small>Existing Archives</small> allows you download, remove and convert archives or view the log of the archiving process: [[File:xmlArchivesNew.png|border|700px]] The archives are grouped by schema, with DarwinCore archives making up an additional group. You can use the minus or plus symbol next to the group header to expand or collapse all files of that group; for web services publishing dozens of datasets, this might be handy to reduce scrolling on the page. The <small>Log </small> link in the group header will open the log file for the respective archiving process in a separate tab. For each archive file, the number of records stored in this file, the date of creation and the file size are listed. XML archives may have a number printed in the <small>Dropped</small> column, meaning some records have been dropped due to missing mandatory data elements. This doesn't mean the archive is invalid, but you should review the mapping or your source data to find the reasons for these lost records. Clicking the archive name will download the archive file, so it can be saved to disk or opened with an archive viewing tool. Pressing the trash symbol at the very right will delete the archive; if you use the trash symbol in the group header, all archives of this schema/type will be removed. For ABCD 2.06 and HISPID 5 XML archives, you will see an additional <small>DwC</small> link next to the trash symbol. This allows you to selectively convert a single dataset from XML to DarwinCore format. In contrast to using the <small>Create DwC Archive</small> button described below, this will spare the XML archiving of the whole web service and just create a DarwinCore archive for the selected dataset, which will be significantly faster. If you use the link in the group header, all archives of the schema will be converted (again without triggering XML archiving first). ==XML Archives== If you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. The section <small>Create/Update Archives</small> allows you to create new XML archive(s) for each schema mapped. Simply select the schema and press <small>Create XML Archive</small>. As mentioned above, archiving ABCD 2.06 or HISPID 5 is dataset-aware, resulting in one XML archive per dataset published. For all other schemas, a single archive file will be created. The paging size specifies how many records will be bundled into one XML document inside the archive; the default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgressNew.png|border]] You can always cancel the archiving process by pressing the <small>Cancel</small> link in the Status/Messages box. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Existing Archives</small> list, or, if it’s still being processed, as <small>Processing</small> in the Status/Messages box. If the process failed for some reason (or you decided to cancel it), a new box <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create XML archives for one schema at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. DarwinCore archives store only one dataset; so as with ABCD 2.06 and HISPID 5 XML archives, DwC-archiving a web service might result in several DarwinCore archives. DarwinCore archives are not created directly, but from the respective ABCD/HISPID archives. So DwC-archiving a web service will first create ACBD archives, which will be converted into the respective DwC archives subsequently. ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the library test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive before, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the archive name listed under ''Existing Archives''. The XML archiving step and the parameter <small>Paging size</small> are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgressNew.png|border]] You can cancel the transformation process by clicking the <small>Cancel</small> link in the Status/Messages box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from the ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: Conversion finished.'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. This parameter corresponds to the paging size that can be set on the archiving page. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. ebd232adbd9f59f1f7c76ef71a8a2fc8130658dd 743 742 2013-03-27T17:28:24Z JoergHoletschek 3 /* Managing Archives */ wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later.<br>If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousand or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The Global Biodiversity Information Facility (GBIF), for example, can use this method for fast indexing of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==Managing Archives== The section <small>Existing Archives</small> allows you download, remove and convert archives or view the log of the archiving process: [[File:xmlArchivesNew.png|border|775px]] The archives are grouped by schema, with DarwinCore archives making up an additional group. You can use the minus or plus symbol next to the group header to expand or collapse all files of that group; for web services publishing dozens of datasets, this might be handy to reduce scrolling on the page. The <small>Log </small> link in the group header will open the log file for the respective archiving process in a separate tab. For each archive file, the number of records stored in this file, the date of creation and the file size are listed. XML archives may have a number printed in the <small>Dropped</small> column, meaning some records have been dropped due to missing mandatory data elements. This doesn't mean the archive is invalid, but you should review the mapping or your source data to find the reasons for these lost records. Clicking the archive name will download the archive file, so it can be saved to disk or opened with an archive viewing tool. Pressing the trash symbol at the very right will delete the archive; if you use the trash symbol in the group header, all archives of this schema/type will be removed. For ABCD 2.06 and HISPID 5 XML archives, you will see an additional <small>DwC</small> link next to the trash symbol. This allows you to selectively convert a single dataset from XML to DarwinCore format. In contrast to using the <small>Create DwC Archive</small> button described below, this will spare the XML archiving of the whole web service and just create a DarwinCore archive for the selected dataset, which will be significantly faster. If you use the link in the group header, all archives of the schema will be converted (again without triggering XML archiving first). ==XML Archives== If you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. The section <small>Create/Update Archives</small> allows you to create new XML archive(s) for each schema mapped. Simply select the schema and press <small>Create XML Archive</small>. As mentioned above, archiving ABCD 2.06 or HISPID 5 is dataset-aware, resulting in one XML archive per dataset published. For all other schemas, a single archive file will be created. The paging size specifies how many records will be bundled into one XML document inside the archive; the default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgressNew.png|border]] You can always cancel the archiving process by pressing the <small>Cancel</small> link in the Status/Messages box. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Existing Archives</small> list, or, if it’s still being processed, as <small>Processing</small> in the Status/Messages box. If the process failed for some reason (or you decided to cancel it), a new box <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create XML archives for one schema at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. DarwinCore archives store only one dataset; so as with ABCD 2.06 and HISPID 5 XML archives, DwC-archiving a web service might result in several DarwinCore archives. DarwinCore archives are not created directly, but from the respective ABCD/HISPID archives. So DwC-archiving a web service will first create ACBD archives, which will be converted into the respective DwC archives subsequently. ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the library test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive before, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the archive name listed under ''Existing Archives''. The XML archiving step and the parameter <small>Paging size</small> are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgressNew.png|border]] You can cancel the transformation process by clicking the <small>Cancel</small> link in the Status/Messages box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from the ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: Conversion finished.'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. This parameter corresponds to the paging size that can be set on the archiving page. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. 257d7da322689280394ccb547210081f7428d9aa 6 109 730 2013-03-27T15:55:49Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 8 2 744 275 2013-03-28T12:24:28Z JoergHoletschek 3 css text/css /* CSS placed here will be applied to all skins */ /* Hide title of Main_Page */ /* body.page-Main_Page h1.firstHeading { display:none; } */ /* Hide every title except special pages */ /*body.ns-subject h1.firstHeading { display:none; }*/ /* Added manually: Hide toolbox */ #p-tb{ display:none;} /* Added manually: Move BreadcrumbsTrail below Heading */ #BreadCrumbsTrail { margin-bottom: -30px; position: relative; /*top: -70px;*/ top: 5px; } #firstHeading { margin-top: 20px; } /* Codeboxen */ div.mw-geshi { padding: 1em; margin: 1em 0; border: 1px dashed #2f6fab; overflow:auto; font-size: 10pt; } 0641c49c4e389643cb28f933d7b56f74faec65a2 745 744 2013-03-28T12:25:31Z JoergHoletschek 3 css text/css /* CSS placed here will be applied to all skins */ /* Hide title of Main_Page */ /* body.page-Main_Page h1.firstHeading { display:none; } */ /* Hide every title except special pages */ /*body.ns-subject h1.firstHeading { display:none; }*/ /* Added manually: Hide toolbox */ #p-tb{ display:none;} /* Added manually: Move BreadcrumbsTrail below Heading */ #BreadCrumbsTrail { margin-bottom: -30px; position: absolute; top: -70px; } #firstHeading { margin-top: 20px; } /* Codeboxen */ div.mw-geshi { padding: 1em; margin: 1em 0; border: 1px dashed #2f6fab; overflow:auto; font-size: 10pt; } f45290d85faaf189859204b00aeb1cbc83c7a56d 746 745 2013-03-28T12:27:00Z JoergHoletschek 3 css text/css /* CSS placed here will be applied to all skins */ /* Hide title of Main_Page */ /* body.page-Main_Page h1.firstHeading { display:none; } */ /* Hide every title except special pages */ /*body.ns-subject h1.firstHeading { display:none; }*/ /* Added manually: Hide toolbox */ #p-tb{ display:none;} /* Added manually: Move BreadcrumbsTrail below Heading */ #BreadCrumbsTrail { margin-bottom: -30px; position: absolute; top: 5px; /*position: relative; top: -70px;*/ } #firstHeading { margin-top: 20px; } /* Codeboxen */ div.mw-geshi { padding: 1em; margin: 1em 0; border: 1px dashed #2f6fab; overflow:auto; font-size: 10pt; } 6993d9724160f0ec7d03fdff44dddb899b733013 0 23 747 711 2013-04-04T15:22:17Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 0e77cafb6fcaa02819dd1b4ddd2c464db1df7b71 748 747 2013-05-29T12:23:22Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 1150153c6bd7c2dae3c637fcb569598cd3c1f1a0 749 748 2013-06-12T09:05:54Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Namespace prefixes added for DNA and EFG extensions. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 9f27d85aa587fc94a34ec86746cd2627be73813b 750 749 2013-07-03T15:03:10Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * XML Archiving can now be limited to certain datasets with one data source. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 02afec4933348e99c830d99082cc05af1bb1beb1 751 750 2013-07-04T12:41:41Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * XML Archiving can now be limited to certain datasets with one data source. * Optional attribute expiration date for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool ac2918c73a531b6c4777ea6c5d3a89db6fd80986 752 751 2013-07-08T12:26:11Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute expiration date for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool e47088ceb432eeee6fc2a83a9bf47ea51b2bcb83 753 752 2013-07-08T13:48:43Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute expiration date for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 6e6c1e0ffdb5f10d52fc9c16d9971f65e6430403 754 753 2013-07-10T13:01:15Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool ac2b7cd84b478333334c019f85545e06e95390b6 755 754 2013-07-15T12:35:16Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 8f0d89548923e61c91562adc7988c0ad46d117df 756 755 2013-09-25T13:56:06Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Concept trees can now be expanded/collapsed in mapping editor (thanks to Falko Glöckler). === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool e6917b6f0c786cfe42c3612c997e66a9d1a7c94d 757 756 2013-10-02T16:39:37Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Bugfix for DarwinCore archives: term recordedBy is now filled correctly. * Concept trees can now be expanded/collapsed in mapping editor (thanks to Falko Glöckler). === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 4d04a1aca411c9542a491546b6dfe41ea68df4a8 758 757 2013-10-02T16:40:00Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Bugfix for DarwinCore archives: term ''recordedBy'' is now filled correctly. * Concept trees can now be expanded/collapsed in mapping editor (thanks to Falko Glöckler). === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 9a3481dd4a3b38a3838754581b0f1ed6225d5d3a 759 758 2013-10-03T18:57:30Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Bugfix for DarwinCore archives: term ''recordedBy'' is now filled correctly. * Mapping editor: ** Bugfix for concept tree (attributes misplaced when ''show all concepts'' was checked) ** Concept tree overhauled (collapsable subtrees thanks to Falko Glöckler) === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 8e942c638e3ffae0816925c4cf09a704caa6c043 760 759 2013-10-04T09:42:54Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Bugfix for DarwinCore archives: term ''recordedBy'' is now filled correctly. * Mapping editor concept tree ** overhauled (collapsable subtrees thanks to Falko Glöckler) ** bugfix for attributes (used to be misplaced when ''show all concepts'' was checked) === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 6ada0fe94b15c10ba0380bf4528d2f918fae457c 761 760 2013-10-04T10:11:52Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Mapping editor concept tree ** overhauled (collapsable subtrees thanks to Falko Glöckler) ** fix for attributes (used to be misplaced when ''show all concepts'' was checked) * Fixes ** term ''recordedBy'' is now filled correctly in DarwinCore archives ** mapping editor now works even if there is no connection to the database === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool db654b853a8798082e9fbae8698f880665550b4e 762 761 2013-10-22T10:09:11Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 2e662fc23c8579980e0f4be7ad9cc4a1ec902b95 763 762 2013-10-22T10:24:48Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 4b6bdbcfd06212a1935c1caed57eba974edb344a 764 763 2013-10-23T11:31:59Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool e2f059f99724fcd2646fc51976ff84b4e0440d51 765 764 2013-10-24T10:16:26Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool a1e16abae3420341962110fb5eafd23cb24101b0 766 765 2013-11-28T14:55:20Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool e70a7ea945bc27689ba771ce5b9cb7aab8ecb1e6 767 766 2013-11-29T15:48:41Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool caa184167cb7fe0436da375db5c1d07aabab465c 769 767 2013-12-05T10:56:46Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 7f49ee9648b5c51e89587dc50f744e2f5dd69267 770 769 2014-01-16T10:38:55Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * DwC Archives ** Terms ''coordinatePrecision'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''). === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 31bf3939a28c73703f1f2eadb5257656f8bf4126 0 52 768 699 2013-11-29T15:59:01Z JoergHoletschek 3 /* Installation */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I'm using Internet Information Server, and after creating the Virtual Directory for BioCASe, I get a “502.2 - Bad Gateway. The specified CGI application misbehaved by not returning a complete set of HTTP headers” error.=== This happens when you didn't correctly associate .cgi and .py files with Python. Make sure you've linked both files with Python and that the path contains the placeholders ''%s %s'' (see [[Installation#Internet_Information_Server|Linking IIS]]). ===I cannot find the installation package of the database connectivity module for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later of psycopg2. ===I'm trying to use BioCASe with Postgres; for this, I tried to install psycopg on Windows. But it doesn't work - either the install crashes, or it finishes, but the test libs page still reports psycopg2 as not installed. In the debug log I can find the error message "ImportError: DLL load failed (...)". What's wrong?=== We suspect this is caused by a bug in recent psycopg2 releases (find more [http://psycopg.lighthouseapp.com/projects/62710/tickets/20 here]). On our test installations, we managed to get round this issue by installing pycopg2 with administration rights. Do do this, right-click the installer file and choose "Run as administrator". ===What's the difference between the DBMS packages odbc_tsql and odbc_tsql_2012 and between pymssql/pymssql_2012, respectively?=== The dbmods ending in "2012" use the new OFFSET clause that was introduced in MS SQL Server 2012. This will make paging faster and increase performance for larger datasets. They work only with SQL Server 2012 or later, and if you use SQL Server 2012, you should use them for best performance. Dbmods without a trailing "2012" work for both older versions and SQL Server 2012, but will be slower in paging large datasets. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== See [[Installation#Updating_an_existing_BioCASe_Installation|Updating an existing BioCASe Installation]]. ===I've updated my installation from a version prior to 2.6. Now, when I enter the DB structure, some of the table and column names are gone, the same happens in the mapping editor.=== Starting with version 2.6, BioCASe retrieves table and column names from the database for some database management systems (namely MySQL, SQL Server, Postgres, Foxpro, Access and Oracle). In older versions, you had to enter table/column names manually, and when you did the setup with such an early version, you probably didn't care for upper/lower cases. Since some DB management systems are case-sensitive, BioCASe can't ignore the differences and won't find these technically incorrect names in the lists retrieved from the database. The respective drop down boxes will therefore be empty. If that happens, just select the correct names from the drop down lists. ===When I try to create an archive, I get a "Permission denied" error. Why is that?=== For creating archives, you need to grant write permissions for the folders ''archive'' and ''www/downloads'' of your BioCASe installation to Python. You can check if permissions are set correctly in the test libs page (Start --> Utilities --> Test Libs). == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other only by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===I've mapped ABCD2. When testing the web service, I get a negative number for "RecordsDropped" in the response document. What does that mean?=== Same problem as above - it means you have an error in your mapping. Most probably you've mapped a non-repeatable ABCD concept to a table/column in your database that holds several entries per unit (occurrence) record. The BioCASe Provider Software will duplicate the respective unit records to be able to create valid ABCD documents - which is indicated by a negative number of "records dropped". Make sure the cardinality of your database relations match the structure (in respects of element repeatability) of ABCD to avoid this problem. ===I'm using pymssql to connect to a SQL Server database. Char(n) and Varchar(n) strings get truncated to 255 characters - what's wrong?=== This is known limitation of TDS protocol used by pymssql. Just change the data type for columns storing long string to TEXT or create a view that converts Char/Varchar columns to TEXT columns on the fly to work around this issue. ===I'm using pymssql to connect to a SQL Server database, BioCASe is running on a Linux distribution. When I do a Search or Scan request, I don't get any results (0 hits), even though there are definitely records in the database.=== That problem occurs on old versions of pymssql (1.0.2 or earlier) on Linux distributions. Update to version 2.0 or later to solve this. If you still get empty result sets, these posts might be helpful: * http://open-up.cybertaxonomy.africamuseum.be/forum_topic/important_if_you_use_biocase_bps_linux_platform_connect_ms_sql_server * http://code.google.com/p/pymssql/issues/detail?id=76. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===I am using BioCASe on a Windows Server. For some queries with the Local QueryForm, I get an error page that is obviously produced by the web server and not BioCASe. It reads "Error Code: 500 Internal Server Error. The request was rejected by the HTTP filter. Contact the server administrator. (12217)"=== This happens when BioCASe is run behind a Microsoft ISA Server/Microsoft Forefront Threat Management Gateway. To solve this problem, follow the instructions given in this article: http://support.microsoft.com/kb/837865. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. == Trouble Shooting == ===I’ve forgotten the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. e354ef74febf202c6e9d645cf9bf6b6d366150a7 0 23 771 770 2014-01-16T11:36:50Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * DwC Archives ** Terms ''coordinatePrecision'', ''georeferenceProtocol'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''). === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool bffdc956953acb1e95bfc67f4b009c57144dd0a8 772 771 2014-01-16T11:52:54Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * DwC Archives ** Terms ''coordinatePrecision'', ''georeferenceProtocol'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''). === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 041cb92db6e27f806da847624f72f4c62905c9bb 773 772 2014-01-16T16:38:01Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * DwC Archives ** Terms ''coordinatePrecision'', ''georeferenceProtocol'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as fieldNumber). ** ABCD element CollectorsFieldNumber is now correctly mapped to recordNumber; fieldNumber is filled with ABCD elemen Gathering/Code. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 331602a54c7ddc2b7f3e43ef4e799a5ebae044e5 774 773 2014-01-16T16:38:26Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * DwC Archives ** Terms ''coordinatePrecision'', ''georeferenceProtocol'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to recordNumber; fieldNumber is filled with ABCD elemen Gathering/Code. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 3e70c05c739b881a7e37cac4a8d958a971e519d4 775 774 2014-01-16T17:02:35Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to recordNumber; fieldNumber is filled with ABCD elemen Gathering/Code. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool b9c9f9ceb0e5d02adb496147d388c799cb29a9c1 776 775 2014-01-16T17:39:04Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to recordNumber; fieldNumber is filled with ABCD elemen Gathering/Code. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 4ca602a31401d269cf9b93e9179bdd782488114a 777 776 2014-01-16T17:39:49Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 3e93c0733908cc7a1a4e4aaf41fa5cb5c44b2132 778 777 2014-01-17T15:41:11Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool c52a22e8db58792d6f6201a0f1243e2e2402da28 779 778 2014-01-23T14:09:10Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 32126234b17da1f2533a53dd949b7f58d3416431 780 779 2014-01-28T09:37:33Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird module revised; works now with current Firebird installation and supports efficient paging. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 64603e36364a2b945491de1c80ed910fae94df3f 781 780 2014-01-28T13:17:51Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird module revised; works now with current Firebird installation and supports efficient paging. * QueryForm now also shows Dataset Inventory. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 706f8bcd4c981a03401c9aafdbb1940d9a07dcdd 783 781 2014-02-04T17:01:43Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird module revised; works now with current Firebird installation and supports efficient paging. ** When Postgres is used, another port than the default 5432 can be used. * QueryForm now also shows Dataset Inventory. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool e3aa5675df58df211cb15749ebd9adb3524f4273 784 783 2014-02-12T11:00:07Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird module revised; works now with current Firebird installation and supports efficient paging. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool cfb47742ea5dc93357e20218060c1a9d3ff5195b 786 784 2014-03-25T16:22:20Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird module revised; works now with current Firebird installation and supports efficient paging. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 53673a4e2a111ed2b6b1704d85f05cf6c85eab35 787 786 2014-03-25T19:58:36Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird module revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 97dbd6b180c335182a327af099d718876ad2af53 788 787 2014-04-01T13:23:44Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird module revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 56a61dbb753d0fb3ba3aa60291285827bcf0f84e 789 788 2014-04-01T13:58:46Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird module revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 44cde7eba27b3e5088b6600a2264aa7128b4d4f8 790 789 2014-04-01T14:03:47Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 5d26ec4e6cf1bba4142383f7ca058db817dae7aa 791 790 2014-04-10T14:06:52Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 87383222c838429adb17992f3ad226004201e957 792 791 2014-04-10T15:06:47Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 6351516bc71c9445dafc265914aa43a149139227 793 792 2014-05-14T14:49:31Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Fix: Admin password removed from configuration dump in system administration. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool db4253a9e09a5a5b502276b7d1d917a6a4948de0 794 793 2014-05-22T12:58:24Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Fixes ** Admin password removed from configuration dump in system administration. ** DwC archiving: For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool d4015aac28c6aaf3b188caf52ac5bbc5f2953ca7 795 794 2014-05-22T13:07:01Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Fixes ** Admin password removed from configuration dump in system administration. ** DwC archiving: For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). * Namespace prefix for ABCD 2.1 added. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 8bd8d053714f7bfb2299785fa30f4d0b35fcc74e 796 795 2014-06-05T11:37:40Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Fixes ** Admin password removed from configuration dump in system administration. ** DwC archiving: For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool e943d9b149f91362b7c230652f2cee7d1ae84ba2 798 796 2014-07-02T14:26:24Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Fixes ** Admin password removed from configuration dump in system administration. ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 7e8214abe2a9586ea88a6193a2b3ecb4c8db5c8b 799 798 2014-07-07T10:37:13Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool b525c3ff26d7b7ee171dd4a450da0f742eab6bfb 800 799 2014-07-17T13:23:21Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 6fda7d36ae8d8d79dd6caad87c9507261317d6d4 801 800 2014-07-18T10:55:24Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool b4702791d3ff455982b25cb72af616803ddde946 802 801 2014-07-18T10:55:33Z JoergHoletschek 3 /* Upcoming [2014-07-18] */ wikitext text/x-wiki == Current Version 3 == === 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 7625f12972f93384eadf2099a6698d121df72e19 803 802 2014-07-18T10:55:47Z JoergHoletschek 3 /* 3.5.2 [2014-07-18] */ wikitext text/x-wiki == Current Version 3 == === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 6d9a9e3fe6ff49f85e48c16966b04caea5127260 804 803 2014-07-18T11:12:18Z JoergHoletschek 3 /* Version 3.5.2 [2014-07-18] */ wikitext text/x-wiki == Current Version 3 == === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool dfa9eade98426f51fb1021e5382f082a4024f44a 807 804 2014-12-16T16:52:07Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if mapped to an empty, i.e. NULL field). === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool be293a6cb99b1334abd8b2d4206b54ecf32eeb3a 808 807 2014-12-16T16:53:04Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribite was mapped to an empty, i.e. NULL field). === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool ec6d19687cc6985391b71fa0e7fe806cf6921e8d 809 808 2014-12-16T16:59:20Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 24f2e3cc3560a47fce64f28f86f6463dee621937 810 809 2014-12-22T14:27:22Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions lather than 2.5, for which 4Suite XML is not available. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 5ce6741c2c0cc05ecdf20a5c7989cf32de60fbd4 811 810 2014-12-22T15:44:21Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions lather than 2.5, for which 4Suite XML is not available. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql: In certain cases, SQL stamenets used to return no results without a reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool c1d127ada742af558a34d70be3294abf70b99167 812 811 2014-12-22T15:45:38Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions lather than 2.5, for which 4Suite XML is not available. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql: In certain cases, SQL stamenets used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool cbde3d37f37c31e45ba7517141287381e4fa9fe8 813 812 2014-12-22T15:54:23Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions lather than 2.5, for which 4Suite XML is not available. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL stamenets used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 8f6e81ce4717909d0471e3c63b030ef878d38dbb 814 813 2015-01-06T14:32:21Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions lather than 2.5, for which 4Suite XML is not available. * Slight behaviour changes in mapping editor: Pop-up window can now be closed with Return or Escape. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 60cafd0015dd77263e334df58ea8fa71924000dd 815 814 2015-01-06T17:04:18Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions lather than 2.5, for which 4Suite XML is not available. * Slight changes in mapping editor (Pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool ceb9d456e2d21b8f424100b508a76ebb3796507b 816 815 2015-01-06T17:04:37Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * Slight changes in mapping editor (Pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool da02245d4f4ebfc3f7ccdc60905aa22887b9f1e7 817 816 2015-01-06T17:04:59Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * Slight changes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 1bdfa005e6a2ad3ead6357945b8b2d92b28ff10e 818 817 2015-01-14T12:34:11Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC) added. * Slight changes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 9668144bf542332c329a2e4c2e1cce89d4c7c14c 819 818 2015-01-14T12:34:29Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 59f1de8f6dec12881ab326e2bdeaa3b069a96f63 820 819 2015-01-15T13:13:36Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrade now don't lose the static table definition any more. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool ff28c86911b94249a15ca0973d131c331ef3f199 0 52 782 768 2014-01-30T15:43:35Z JoergHoletschek 3 wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I'm using Internet Information Server, and after creating the Virtual Directory for BioCASe, I get a “502.2 - Bad Gateway. The specified CGI application misbehaved by not returning a complete set of HTTP headers” error.=== This happens when you didn't correctly associate .cgi and .py files with Python. Make sure you've linked both files with Python and that the path contains the placeholders ''%s %s'' (see [[Installation#Internet_Information_Server|Linking IIS]]). ===I cannot find the installation package of the database connectivity module for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later of psycopg2. ===I'm trying to use BioCASe with Postgres; for this, I tried to install psycopg on Windows. But it doesn't work - either the install crashes, or it finishes, but the test libs page still reports psycopg2 as not installed. In the debug log I can find the error message "ImportError: DLL load failed (...)". What's wrong?=== We suspect this is caused by a bug in recent psycopg2 releases (find more [http://psycopg.lighthouseapp.com/projects/62710/tickets/20 here]). On our test installations, we managed to get round this issue by installing pycopg2 with administration rights. Do do this, right-click the installer file and choose "Run as administrator". ===What's the difference between the DBMS packages odbc_tsql and odbc_tsql_2012 and between pymssql/pymssql_2012, respectively?=== The dbmods ending in "2012" use the new OFFSET clause that was introduced in MS SQL Server 2012. This will make paging faster and increase performance for larger datasets. They work only with SQL Server 2012 or later, and if you use SQL Server 2012, you should use them for best performance. Dbmods without a trailing "2012" work for both older versions and SQL Server 2012, but will be slower in paging large datasets. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== See [[Installation#Updating_an_existing_BioCASe_Installation|Updating an existing BioCASe Installation]]. ===I've updated my installation from a version prior to 2.6. Now, when I enter the DB structure, some of the table and column names are gone, the same happens in the mapping editor.=== Starting with version 2.6, BioCASe retrieves table and column names from the database for some database management systems (namely MySQL, SQL Server, Postgres, Foxpro, Access and Oracle). In older versions, you had to enter table/column names manually, and when you did the setup with such an early version, you probably didn't care for upper/lower cases. Since some DB management systems are case-sensitive, BioCASe can't ignore the differences and won't find these technically incorrect names in the lists retrieved from the database. The respective drop down boxes will therefore be empty. If that happens, just select the correct names from the drop down lists. ===When I try to create an archive, I get a "Permission denied" error. Why is that?=== For creating archives, you need to grant write permissions for the folders ''archive'' and ''www/downloads'' of your BioCASe installation to Python. You can check if permissions are set correctly in the test libs page (Start --> Utilities --> Test Libs). == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other only by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===I've mapped ABCD2. When testing the web service, I get a negative number for "RecordsDropped" in the response document. What does that mean?=== Same problem as above - it means you have an error in your mapping. Most probably you've mapped a non-repeatable ABCD concept to a table/column in your database that holds several entries per unit (occurrence) record. The BioCASe Provider Software will duplicate the respective unit records to be able to create valid ABCD documents - which is indicated by a negative number of "records dropped". Make sure the cardinality of your database relations match the structure (in respects of element repeatability) of ABCD to avoid this problem. ===I'm using pymssql to connect to a SQL Server database. Char(n) and Varchar(n) strings get truncated to 255 characters - what's wrong?=== This is known limitation of TDS protocol used by pymssql. Just change the data type for columns storing long string to TEXT or create a view that converts Char/Varchar columns to TEXT columns on the fly to work around this issue. ===I'm using pymssql to connect to a SQL Server database, BioCASe is running on a Linux distribution. When I do a Search or Scan request, I don't get any results (0 hits), even though there are definitely records in the database.=== That problem occurs on old versions of pymssql (1.0.2 or earlier) on Linux distributions. Update to version 2.0 or later to solve this. If you still get empty result sets, these posts might be helpful: * http://open-up.cybertaxonomy.africamuseum.be/forum_topic/important_if_you_use_biocase_bps_linux_platform_connect_ms_sql_server * http://code.google.com/p/pymssql/issues/detail?id=76. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===I am using BioCASe on a Windows Server. For some queries with the Local QueryForm, I get an error page that is obviously produced by the web server and not BioCASe. It reads "Error Code: 500 Internal Server Error. The request was rejected by the HTTP filter. Contact the server administrator. (12217)"=== This happens when BioCASe is run behind a Microsoft ISA Server/Microsoft Forefront Threat Management Gateway. To solve this problem, follow the instructions given in this article: http://support.microsoft.com/kb/837865. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. ===I'm using BioCASe in a Vagrant Virtual Box on MacOS with Synced Folders. When using the Manual Query Form or the Archiving page, I get an "ImportError: No module named wrapper.errorclasses."=== We don't know exactly what causes this, but apparently it happens when BioCASe is run in a Vagrant Virtual Box on MacOS and Vagrant's synced folder are used. We do have a workaround for this, but it is not easy. Please [[Special:Contact|contact us]] for assistance on this. == Trouble Shooting == ===I’ve forgotten the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. d5ff7278462c3a560d5aaab2c7c42eff029a703d 785 782 2014-02-12T11:07:17Z JoergHoletschek 3 /* Installation */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I'm using Internet Information Server, and after creating the Virtual Directory for BioCASe, I get a “502.2 - Bad Gateway. The specified CGI application misbehaved by not returning a complete set of HTTP headers” error.=== This happens when you didn't correctly associate .cgi and .py files with Python. Make sure you've linked both files with Python and that the path contains the placeholders ''%s %s'' (see [[Installation#Internet_Information_Server|Linking IIS]]). ===I cannot find the installation package of the database connectivity module for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later of psycopg2. ===I'm trying to use BioCASe with Postgres; for this, I tried to install psycopg on Windows. But it doesn't work - either the install crashes, or it finishes, but the test libs page still reports psycopg2 as not installed. In the debug log I can find the error message "ImportError: DLL load failed (...)". What's wrong?=== We suspect this is caused by a bug in recent psycopg2 releases (find more [http://psycopg.lighthouseapp.com/projects/62710/tickets/20 here]). On our test installations, we managed to get round this issue by installing pycopg2 with administration rights. Do do this, right-click the installer file and choose "Run as administrator". ===What's the difference between the DBMS packages odbc_tsql and odbc_tsql_2012 and between pymssql/pymssql_2012, respectively?=== The dbmods ending in "2012" use the new OFFSET clause that was introduced in MS SQL Server 2012. This will make paging faster and increase performance for larger datasets. They work only with SQL Server 2012 or later, and if you use SQL Server 2012, you should use them for best performance. Dbmods without a trailing "2012" work for both older versions and SQL Server 2012, but will be slower in paging large datasets. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== See [[Installation#Updating_an_existing_BioCASe_Installation|Updating an existing BioCASe Installation]]. ===I've updated my installation from a version prior to 2.6. Now, when I enter the DB structure, some of the table and column names are gone, the same happens in the mapping editor.=== Starting with version 2.6, BioCASe retrieves table and column names from the database for some database management systems (namely MySQL, SQL Server, Postgres, Foxpro, Access and Oracle). In older versions, you had to enter table/column names manually, and when you did the setup with such an early version, you probably didn't care for upper/lower cases. Since some DB management systems are case-sensitive, BioCASe can't ignore the differences and won't find these technically incorrect names in the lists retrieved from the database. The respective drop down boxes will therefore be empty. If that happens, just select the correct names from the drop down lists. ===When I try to create an archive, I get a "Permission denied" error. Why is that?=== For creating archives, you need to grant write permissions for the folders ''archive'' and ''www/downloads'' of your BioCASe installation to Python. You can check if permissions are set correctly in the test libs page (Start --> Utilities --> Test Libs). ===I'm trying to install BioCASe on Ubuntu (or another Linux). Even though I've run the setup script and added the snippet to the Apache configuration file, I see source code when I try out my installation instead of the pages described in the documentation=== Apparently CGI is not enabled on your machine. Google how to enable CGI for your distribution and follow the instructions you'll find. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other only by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===I've mapped ABCD2. When testing the web service, I get a negative number for "RecordsDropped" in the response document. What does that mean?=== Same problem as above - it means you have an error in your mapping. Most probably you've mapped a non-repeatable ABCD concept to a table/column in your database that holds several entries per unit (occurrence) record. The BioCASe Provider Software will duplicate the respective unit records to be able to create valid ABCD documents - which is indicated by a negative number of "records dropped". Make sure the cardinality of your database relations match the structure (in respects of element repeatability) of ABCD to avoid this problem. ===I'm using pymssql to connect to a SQL Server database. Char(n) and Varchar(n) strings get truncated to 255 characters - what's wrong?=== This is known limitation of TDS protocol used by pymssql. Just change the data type for columns storing long string to TEXT or create a view that converts Char/Varchar columns to TEXT columns on the fly to work around this issue. ===I'm using pymssql to connect to a SQL Server database, BioCASe is running on a Linux distribution. When I do a Search or Scan request, I don't get any results (0 hits), even though there are definitely records in the database.=== That problem occurs on old versions of pymssql (1.0.2 or earlier) on Linux distributions. Update to version 2.0 or later to solve this. If you still get empty result sets, these posts might be helpful: * http://open-up.cybertaxonomy.africamuseum.be/forum_topic/important_if_you_use_biocase_bps_linux_platform_connect_ms_sql_server * http://code.google.com/p/pymssql/issues/detail?id=76. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===I am using BioCASe on a Windows Server. For some queries with the Local QueryForm, I get an error page that is obviously produced by the web server and not BioCASe. It reads "Error Code: 500 Internal Server Error. The request was rejected by the HTTP filter. Contact the server administrator. (12217)"=== This happens when BioCASe is run behind a Microsoft ISA Server/Microsoft Forefront Threat Management Gateway. To solve this problem, follow the instructions given in this article: http://support.microsoft.com/kb/837865. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. ===I'm using BioCASe in a Vagrant Virtual Box on MacOS with Synced Folders. When using the Manual Query Form or the Archiving page, I get an "ImportError: No module named wrapper.errorclasses."=== We don't know exactly what causes this, but apparently it happens when BioCASe is run in a Vagrant Virtual Box on MacOS and Vagrant's synced folder are used. We do have a workaround for this, but it is not easy. Please [[Special:Contact|contact us]] for assistance on this. == Trouble Shooting == ===I’ve forgotten the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. 6dbd1e218ab4c50911065bd998e89552c2c368c5 797 785 2014-07-01T13:20:00Z JoergHoletschek 3 /* Debugging */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/informatics/infrastructure/publishing/) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I'm using Internet Information Server, and after creating the Virtual Directory for BioCASe, I get a “502.2 - Bad Gateway. The specified CGI application misbehaved by not returning a complete set of HTTP headers” error.=== This happens when you didn't correctly associate .cgi and .py files with Python. Make sure you've linked both files with Python and that the path contains the placeholders ''%s %s'' (see [[Installation#Internet_Information_Server|Linking IIS]]). ===I cannot find the installation package of the database connectivity module for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later of psycopg2. ===I'm trying to use BioCASe with Postgres; for this, I tried to install psycopg on Windows. But it doesn't work - either the install crashes, or it finishes, but the test libs page still reports psycopg2 as not installed. In the debug log I can find the error message "ImportError: DLL load failed (...)". What's wrong?=== We suspect this is caused by a bug in recent psycopg2 releases (find more [http://psycopg.lighthouseapp.com/projects/62710/tickets/20 here]). On our test installations, we managed to get round this issue by installing pycopg2 with administration rights. Do do this, right-click the installer file and choose "Run as administrator". ===What's the difference between the DBMS packages odbc_tsql and odbc_tsql_2012 and between pymssql/pymssql_2012, respectively?=== The dbmods ending in "2012" use the new OFFSET clause that was introduced in MS SQL Server 2012. This will make paging faster and increase performance for larger datasets. They work only with SQL Server 2012 or later, and if you use SQL Server 2012, you should use them for best performance. Dbmods without a trailing "2012" work for both older versions and SQL Server 2012, but will be slower in paging large datasets. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== See [[Installation#Updating_an_existing_BioCASe_Installation|Updating an existing BioCASe Installation]]. ===I've updated my installation from a version prior to 2.6. Now, when I enter the DB structure, some of the table and column names are gone, the same happens in the mapping editor.=== Starting with version 2.6, BioCASe retrieves table and column names from the database for some database management systems (namely MySQL, SQL Server, Postgres, Foxpro, Access and Oracle). In older versions, you had to enter table/column names manually, and when you did the setup with such an early version, you probably didn't care for upper/lower cases. Since some DB management systems are case-sensitive, BioCASe can't ignore the differences and won't find these technically incorrect names in the lists retrieved from the database. The respective drop down boxes will therefore be empty. If that happens, just select the correct names from the drop down lists. ===When I try to create an archive, I get a "Permission denied" error. Why is that?=== For creating archives, you need to grant write permissions for the folders ''archive'' and ''www/downloads'' of your BioCASe installation to Python. You can check if permissions are set correctly in the test libs page (Start --> Utilities --> Test Libs). ===I'm trying to install BioCASe on Ubuntu (or another Linux). Even though I've run the setup script and added the snippet to the Apache configuration file, I see source code when I try out my installation instead of the pages described in the documentation=== Apparently CGI is not enabled on your machine. Google how to enable CGI for your distribution and follow the instructions you'll find. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other only by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===I've mapped ABCD2. When testing the web service, I get a negative number for "RecordsDropped" in the response document. What does that mean?=== Same problem as above - it means you have an error in your mapping. Most probably you've mapped a non-repeatable ABCD concept to a table/column in your database that holds several entries per unit (occurrence) record. The BioCASe Provider Software will duplicate the respective unit records to be able to create valid ABCD documents - which is indicated by a negative number of "records dropped". Make sure the cardinality of your database relations match the structure (in respects of element repeatability) of ABCD to avoid this problem. ===I'm using pymssql to connect to a SQL Server database. Char(n) and Varchar(n) strings get truncated to 255 characters - what's wrong?=== This is known limitation of TDS protocol used by pymssql. Just change the data type for columns storing long string to TEXT or create a view that converts Char/Varchar columns to TEXT columns on the fly to work around this issue. ===I'm using pymssql to connect to a SQL Server database, BioCASe is running on a Linux distribution. When I do a Search or Scan request, I don't get any results (0 hits), even though there are definitely records in the database.=== That problem occurs on old versions of pymssql (1.0.2 or earlier) on Linux distributions. Update to version 2.0 or later to solve this. If you still get empty result sets, these posts might be helpful: * http://open-up.cybertaxonomy.africamuseum.be/forum_topic/important_if_you_use_biocase_bps_linux_platform_connect_ms_sql_server * http://code.google.com/p/pymssql/issues/detail?id=76. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===I am using BioCASe on a Windows Server. For some queries with the Local QueryForm, I get an error page that is obviously produced by the web server and not BioCASe. It reads "Error Code: 500 Internal Server Error. The request was rejected by the HTTP filter. Contact the server administrator. (12217)"=== This happens when BioCASe is run behind a Microsoft ISA Server/Microsoft Forefront Threat Management Gateway. To solve this problem, follow the instructions given in this article: http://support.microsoft.com/kb/837865. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. ===I'm using BioCASe in a Vagrant Virtual Box on MacOS with Synced Folders. When using the Manual Query Form or the Archiving page, I get an "ImportError: No module named wrapper.errorclasses."=== We don't know exactly what causes this, but apparently it happens when BioCASe is run in a Vagrant Virtual Box on MacOS and Vagrant's synced folder are used. We do have a workaround for this, but it is not easy. Please [[Special:Contact|contact us]] for assistance on this. ===I'm using BioCASe on a Windows server and odbc_tsql to connect to an MS SQL Server. Even though I do get results for a SCAN request, I cannot do a SEARCH (invalid document received).=== Please check your encoding settings in the DB connection configuration. If it is set to UTF_8, try Latin_1 instead. == Trouble Shooting == ===I’ve forgotten the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. bd1eea279a5c3739fdab9c2fa45f54ce1fb4158b 0 6 805 591 2014-09-18T09:21:07Z JoergHoletschek 3 /* Updating an existing BioCASe Installation */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html === Testing the Link === Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages/Java == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! === Updating from version 2.5 or later === # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. === Updating from older versions < 2.5 === # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. dd1ede66c4edc88e8430b9ee726037509d0f2fb2 806 805 2014-09-18T09:22:24Z JoergHoletschek 3 /* Updating an existing BioCASe Installation */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. Recommended Apache versions are 2.0 and 2.2. The BPS requires Python 2.5+ to run. Even though it also works with versions 2.6 and 2.7, the recommended Python version is 2.5. For later versions, the package required for connecting the BPS to the database used may be not available yet. You can go to the [http://ww3.bgbm.org/biocase/utilities/testlibs.cgi libs test page of the BGBM installation], find the package required for your DBMS and check out the availability for newer Python versions on the respective website (for example, the 4SuiteXML library required for running the Local Query Tool is currently only available for Python 2.5). By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html === Testing the Link === Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages/Java == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! === Updating from version 2.5 or later === # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. === Updating from older versions < 2.5 === # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. 4fbd71a8b457a8ef796d54739371418b60385cd9 0 23 821 820 2015-01-15T13:15:05Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 2c6a09bc0dddf82e7426b75926bfa66c80040159 822 821 2015-01-15T17:04:37Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 37ab2f9f9c7e6f177f70db5475eaad7abc644eb5 829 822 2015-01-23T15:51:33Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 42dc4eec94c1873193d0a43f2b44c9a26d543e26 830 829 2015-01-23T15:51:45Z JoergHoletschek 3 /* 3.5.3 [2015-01-23] */ wikitext text/x-wiki == Current Version 3 == === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool a8a050eb8ed8f525f035e31fb1c2e03ee8ef6c71 831 830 2015-04-01T15:02:32Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Fixes ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool c2a010cb68c23f1f6244cb940ec18a4af6798d56 832 831 2015-04-03T13:20:54Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Smaller changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * Fixes ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool f9b7af556ab500dcf1d7722c1f983ac09fef7657 833 832 2015-06-19T08:27:54Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * Fixes ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 4d5f87d71dbb00bd019cc2b2807472631da4bf26 843 833 2015-08-10T15:26:48Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * Fixes ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool c49cec760e26378231f543111a45b75b8488d2ad 844 843 2015-08-14T12:30:44Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * Support for ABCD 2.1 added to QueryTool and archiving. * Fixes ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 9800119c6040b2f3dd6b3c00dd0221667a7c0c19 845 844 2015-08-14T14:52:55Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * Support for ABCD 2.1 added to QueryTool and archiving. * Fixes ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 3a958bf235f416bd5fa96d1b987f892d144113a9 846 845 2015-08-18T15:01:20Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * Support for ABCD 2.1 added to QueryTool and archiving. * Fixes ** Namespace added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool ef972d4fd019b9c4190323001251276b8131cb70 848 846 2015-08-31T10:14:26Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * Support for ABCD 2.1 added to QueryTool and archiving. * Support for LIDO schema added (including smaller changes to wrapper core). * Fixes ** Namespace added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 7c08121d05ffa6b5aaa38d2761f76e054156da31 849 848 2015-09-06T15:53:43Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * Support for ABCD 2.1 added to QueryTool and archiving. * Support for LIDO schema added (including smaller changes to wrapper core). * The ''schemaLocation'' is now set for some schemas. * Fixes ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 8967793b3931901ca9d0c8edfabb86f57afc66d9 850 849 2015-09-07T16:22:33Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * Support for ABCD 2.1 added to QueryTool and archiving. * Support for LIDO schema added (including smaller changes to wrapper core). * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * Fixes ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 795cf7b4553a8066f24dbcf773d4fce585a7ed22 851 850 2015-09-15T10:01:43Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * Support for ABCD 2.1 added to QueryTool and archiving. * Support for LIDO schema added (including smaller changes to wrapper core). * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * Fixes ** DarwinCore archive transformation: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 1ffd3350acc67e1918b8007d4d7ac1f768b9131c 852 851 2015-09-17T12:34:25Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Support for [http://http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * Fixes ** DarwinCore archive transformation: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 234342efa77b8fba9452c69a64dd978d22f542ae 853 852 2015-09-17T12:34:48Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * Fixes ** DarwinCore archive transformation: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 6118f35bec4e032390ae58449ab25daf16b30024 854 853 2015-10-09T10:04:12Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * Fixes ** DarwinCore archive transformation: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 8add686abcc4387f417ee0c9c1ffbe9ca048d22a 855 854 2015-10-09T10:09:58Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore archive transformation: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool f21cf28b6d0a2447a3a260c751c7e549494a215e 856 855 2015-10-09T10:34:21Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore archive transformation: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 60c15afd911450085b4f115f4a26cb2e457ce69a 857 856 2015-10-14T15:49:49Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * Support for ABCD 2.1 added to QueryTool and archiving. * XML Archiving will be more patient now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** XML archiving uses less memory now (and will be probably faster with larger datasets) ** DarwinCore archive transformation: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool e935ee2455c84e7a841971eded802e7a8ad14902 858 857 2015-10-19T13:13:09Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore archive transformation: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 20c0aa3e092771a06603dc246ad95ee6c2a02c31 859 858 2015-10-27T14:12:33Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore archive transformation: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 527f4155c9a6e4fdb6d8a5b43b6571e2f3f56473 860 859 2015-11-09T11:27:23Z JoergHoletschek 3 wikitext text/x-wiki == Current Version 3 == === Upcoming === * Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore archive transformation: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 65fa1ed2b6383d158d6cb5698ec39d2833c206cd 861 860 2015-11-09T11:27:58Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * DwC Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore archive transformation: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 7e0a9ec816d43d635200882e98c65b6bc96474ea 862 861 2015-11-09T14:39:33Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool fd28e7269fa174e0e8d40b3eda92856b80efe505 869 862 2015-11-18T16:40:05Z JoergHoletschek 3 wikitext text/x-wiki == Current Version 3 == === Upcoming === * New dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 39e3934ee58457956912da659f9b2e709fb1ff43 870 869 2015-11-19T13:29:28Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * New dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 2d955f5f1af35068f362ff284562102ae97c33f1 0 110 823 2015-01-19T14:47:51Z JoergHoletschek 3 Created page with "Was ist die BioCASe Provider Software? Die BioCASe Provider Software ist eine Middleware, also eine Software, die den Datenaustausch zwischen zwei voneinander entfernten Syste..." wikitext text/x-wiki Was ist die BioCASe Provider Software? Die BioCASe Provider Software ist eine Middleware, also eine Software, die den Datenaustausch zwischen zwei voneinander entfernten Systemen erlaubt. Sie ermöglicht, Daten aus einem relationalen Datenbanksystem (z.B. einem Sammlungsverwaltungssystem) in ein Datennetzwerk einzuspeisen. Nach der Installation und Konfiguration von BioCASe sind die Daten als BioCASe Webservice im Internet verfügbar und können mittels BioCASe-Protokollanfragen abgerufen werden. Die Provider Software ist nicht abhängig von einem bestimmten Datenstandard und lässt sich mit einem beliebigen Datenschema verwenden. Die Provider Software muss auf einem Webserver des Datenlieferanten installiert werden. Kern ist der sogenannte PyWrapper, eine in Python entwickelte XML/CGI-Datenschnittstelle, die einen standardisierten Zugriff auf eine Vielzahl von Datenbankmanagementsystemen mit beliebigen Datenmodellen ermöglicht. Um diese Schnittstelle herum existieren verschiedene Tools, die die Einrichtung, Konfiguration und den Test der BioCASe-Installation und –Webservices vereinfachen. Obwohl BioCASe prinzipiell mit jedem XML-Datenschema verwendet werden kann, ist das Haupteinsatzfeld die Anbindung von biologischen Beleg- und Beobachtungsdaten an Primärdatennetzwerke zur Artenvielfalt wie dem BioCASe-Netzwerk und der GBIF. Voraussetzungen für den Einsatz der BioCASe Provider Software Vor dem Einsatz der Provider Software sollte ein potentieller Datenlieferant drei Voraussetzungen prüfen: Die zu veröffentlichenden Sammlungs- oder Beobachtungsdaten müssen in einem SQL-fähigen relationalen Datenbankmanagementsystem vorhanden sein oder in einem Format, zu dem ein Zugang über ODBC (Open Database Connectivity) möglich ist, wie z.B. Microsoft Excel. Nicht-relationale Datenbankmanagementsysteme werden nicht unterstützt. Derzeit kann sich die Provider Software mit den folgenden Datenbanksystemen verbinden: Microsoft Access, Microsoft SQL Server, MySQL, PostgreSQL, Oracle, Foxpro, Sybase, 4D, DB2, Firebird, Microsoft Excel. Zur Installation der Provider Software muss ein dauerhaft mit dem Internet verbundener Webserver vorhanden sein (Apache oder Microsoft IIS). Dies kann ein Server des Datenproviders oder eines kommerziellen Hosting-Dienstes sein. Eine Veröffentlichung über eine Dial-up-Verbindung ist nicht möglich. Auf diesem Webserver muss die Installation von Python (insofern es noch nicht installiert ist, was z.B. bei vielen gängigen Linux-Installationen der Fall ist) sowie zusätzlicher Python-Pakete möglich sein. Dies ist für Mitarbeiter der IT üblicherweise keine Hürde, solange dem keine institutionellen Richtlinien entgegenstehen. Für die optionale Erzeugung von DarwinCore-Archiven ist zusätzlich eine Java Runtime Environment nötig. Vorgehensweise Sollte die Wahl auf die BioCASe Provider Software fallen (bei der Entscheidungsfindung hilft das BioCASe-Team gerne weiter), umfasst eine Anbindung von Daten üblicherweise die folgenden Schritte: • Entscheiden, welche Daten an das Netzwerk angebunden werden sollen. Dazu gehören die Klärung der Frage, wem die Daten eigentlich gehören, sowie die Festlegung der Nutzungsbedingungen/-beschränkungen und einer Lizenz. Darüber hinaus sollte geprüft werden, welche Daten *nicht* veröffentlicht werden sollen, weil sie gefährdete Arten betreffen. In diesem Fall können bestimmte Datensätze komplett von der Veröffentlichung ausgeschlossen werden oder bestimmte Teile davon – z.B. die Koordinaten des Fundortes – gezielt verunschärft werden. • Vorbereitung der Datenbank auf die Veröffentlichung. Dabei müssen die Entscheidung getroffen werden, ob die Live-Datenbank oder ein regelmäßig erzeugter Snapshot angebunden werden soll, und die Tabelle für die Metadaten angelegt und befüllt werden. In einigen Fällen sind weitere Datentransformationen nötig, siehe Tutorial. • Installation der Provider Software, siehe Tutorial. • Anlegen einer BioCASe Datenquelle, Verbinden mit der Datenbank und Definition der Datenbankstruktur, siehe . • Anlegen eines ABCD2 Mappings. • Testen und Debuggen des BioCASe Webservices, siehe Tutorial. • Nach Abschluss der Konfiguration kann der neue BioCASe-Webservice im Wunsch-Datennetzwerk angemeldet werden, z.B. bei GBIF. Wir empfehlen jedoch, davor kurz das BioCASe-Team zu kontaktieren, das einen Blick auf den neuen Webservice werfen und auf Richtigkeit prüfen kann. Falsche Mappings lassen sich umso leichter korrigieren, je früher sie entdeckt werden. Support Bei Problemen während Installation und Konfiguration empfehlen wir • In den FAQ nachzuschlagen, ob das Problem und eine Lösung dort beschrieben sind, • Befreundete Nutzer der BioCASe Provider Software um Rat zu fragen, oder • Kontakt mit dem BioCASe-Team aufzunehmen. Bitte dabei genau das verwendete System (verwendetes BioCASe-Version sowie Betriebs/Datenbankmanagementsystem), Fehlermeldungen und die Schritte, die dazu führten, beschreiben. Screenshots sind dabei immer von Vorteil. c0fbc9db0f3ea11e09a651cdc7f8c78a4e8cbde0 824 823 2015-01-19T14:59:11Z JoergHoletschek 3 wikitext text/x-wiki ==Was ist die BioCASe Provider Software?== Die BioCASe Provider Software ist eine Middleware, also eine Software, die den Datenaustausch zwischen zwei voneinander entfernten Systemen erlaubt. Sie ermöglicht, Daten aus einem relationalen Datenbanksystem (z.B. einem Sammlungsverwaltungssystem) in ein Datennetzwerk einzuspeisen. Nach der Installation und Konfiguration von BioCASe sind die Daten als BioCASe Webservice im Internet verfügbar und können mittels BioCASe-Protokollanfragen abgerufen werden. Die Provider Software ist nicht abhängig von einem bestimmten Datenstandard und lässt sich mit einem beliebigen Datenschema verwenden. Die Provider Software muss auf einem Webserver des Datenlieferanten installiert werden. Kern ist der sogenannte PyWrapper, eine in Python entwickelte XML/CGI-Datenschnittstelle, die einen standardisierten Zugriff auf eine Vielzahl von Datenbankmanagementsystemen mit beliebigen Datenmodellen ermöglicht. Um diese Schnittstelle herum existieren verschiedene Tools, die die Einrichtung, Konfiguration und den Test der BioCASe-Installation und –Webservices vereinfachen. Obwohl BioCASe prinzipiell mit jedem XML-Datenschema verwendet werden kann, ist das Haupteinsatzfeld die Anbindung von biologischen Beleg- und Beobachtungsdaten an Primärdatennetzwerke zur Artenvielfalt wie dem [http://www.biocase.org BioCASe Netzwerk] und der [http://www.gbif.org Global Biodiversity Information Facility]. ==Voraussetzungen für den Einsatz der BioCASe Provider Software== Vor dem Einsatz der Provider Software sollte ein potentieller Datenlieferant drei Voraussetzungen prüfen: * Die zu veröffentlichenden Sammlungs- oder Beobachtungsdaten müssen in einem SQL-fähigen relationalen Datenbankmanagementsystem vorhanden sein oder in einem Format, zu dem ein Zugang über ODBC (Open Database Connectivity) möglich ist, wie z.B. Microsoft Excel. Nicht-relationale Datenbankmanagementsysteme werden nicht unterstützt. Derzeit kann sich die Provider Software mit den folgenden Datenbanksystemen verbinden: Microsoft Access, Microsoft SQL Server, MySQL, PostgreSQL, Oracle, Foxpro, Sybase, 4D, DB2, Firebird, Microsoft Excel. * Zur Installation der Provider Software muss ein dauerhaft mit dem Internet verbundener Webserver vorhanden sein (Apache oder Microsoft IIS). Dies kann ein Server des Datenproviders oder eines kommerziellen Hosting-Dienstes sein. Eine Veröffentlichung über eine Dial-up-Verbindung ist nicht möglich. * Auf diesem Webserver muss die Installation von Python (insofern es noch nicht installiert ist, was z.B. bei vielen gängigen Linux-Installationen der Fall ist) sowie zusätzlicher Python-Pakete möglich sein. Dies ist für Mitarbeiter der IT üblicherweise keine Hürde, solange dem keine institutionellen Richtlinien entgegenstehen. Für die optionale Erzeugung von DarwinCore-Archiven ist zusätzlich eine Java Runtime Environment nötig. ==Vorgehensweise== Sollte die Wahl auf die BioCASe Provider Software fallen (bei der Entscheidungsfindung hilft das [[Special:Contact|BioCASe Team]] gerne weiter), umfasst eine Anbindung von Daten üblicherweise die folgenden Schritte: # Entscheiden, welche Daten an das Netzwerk angebunden werden sollen. Dazu gehören die Klärung der Frage, wem die Daten eigentlich gehören, sowie die Festlegung der Nutzungsbedingungen/-beschränkungen und einer Lizenz. Darüber hinaus sollte geprüft werden, welche Daten '''nicht''' veröffentlicht werden sollen, weil sie gefährdete Arten betreffen. In diesem Fall können bestimmte Datensätze komplett von der Veröffentlichung ausgeschlossen werden oder bestimmte Teile davon – z.B. die Koordinaten des Fundortes – gezielt verunschärft werden. # Vorbereitung der Datenbank auf die Veröffentlichung. Dabei müssen die Entscheidung getroffen werden, ob die Live-Datenbank oder ein regelmäßig erzeugter Snapshot angebunden werden soll, und die Tabelle für die Metadaten angelegt und befüllt werden. In einigen Fällen sind weitere Datentransformationen nötig, siehe [[Preparation|Tutorial]]. # Installation der Provider Software, siehe [[Installation|Tutorial]]. # Anlegen einer BioCASe Datenquelle, Verbinden mit der Datenbank und Definition der Datenbankstruktur, siehe [[DatasourceSetup|Tutorial]]. # Anlegen eines [[ABCD2Mapping|ABCD2 Mappings]]. # Testen und Debuggen des BioCASe Webservices, siehe [[Debugging|Tutorial]]. # Nach Abschluss der Konfiguration kann der neue BioCASe-Webservice im Wunsch-Datennetzwerk angemeldet werden, z.B. bei [http://www.gbif.org GBIF]. Wir empfehlen jedoch, davor kurz das [[Special:Contact|BioCASe-Team zu kontaktieren]], das einen Blick auf den neuen Webservice werfen und auf Richtigkeit prüfen kann. Falsche Mappings lassen sich umso leichter korrigieren, je früher sie entdeckt werden. ==Support== Bei Problemen während Installation und Konfiguration empfehlen wir * In den [[FAQ]]nachzuschlagen, ob das Problem und eine Lösung dort beschrieben sind, * Befreundete Nutzer der BioCASe Provider Software um Rat zu fragen, oder * Kontakt mit dem [[Special:Contact|BioCASe Team]] aufzunehmen. Bitte dabei genau das verwendete System (verwendetes BioCASe-Version sowie Betriebs/Datenbankmanagementsystem), Fehlermeldungen und die Schritte, die dazu führten, beschreiben. Screenshots sind dabei immer von Vorteil. ed040049ab668f28c9ab5cbd5aa69cfadb2d8100 825 824 2015-01-19T15:01:47Z JoergHoletschek 3 wikitext text/x-wiki ==Was ist die BioCASe Provider Software?== Die BioCASe Provider Software ist eine Middleware, also eine Software, die den Datenaustausch zwischen zwei voneinander entfernten Systemen erlaubt. Sie ermöglicht, Daten aus einem relationalen Datenbanksystem (z.B. einem Sammlungsverwaltungssystem) in ein Datennetzwerk einzuspeisen. Nach der Installation und Konfiguration von BioCASe sind die Daten als BioCASe Webservice im Internet verfügbar und können mittels BioCASe-Protokollanfragen abgerufen werden. Die Provider Software ist nicht abhängig von einem bestimmten Datenstandard und lässt sich mit einem beliebigen Datenschema verwenden. Die Provider Software muss auf einem Webserver des Datenlieferanten installiert werden. Kern ist der sogenannte PyWrapper, eine in Python entwickelte XML/CGI-Datenschnittstelle, die einen standardisierten Zugriff auf eine Vielzahl von Datenbankmanagementsystemen mit beliebigen Datenmodellen ermöglicht. Um diese Schnittstelle herum existieren verschiedene Tools, die die Einrichtung, Konfiguration und den Test der BioCASe-Installation und –Webservices vereinfachen. Obwohl BioCASe prinzipiell mit jedem XML-Datenschema verwendet werden kann, ist das Haupteinsatzfeld die Anbindung von biologischen Beleg- und Beobachtungsdaten an Primärdatennetzwerke zur Artenvielfalt wie dem [http://www.biocase.org BioCASe Netzwerk] und der [http://www.gbif.org Global Biodiversity Information Facility]. ==Voraussetzungen für den Einsatz der BioCASe Provider Software== Vor dem Einsatz der Provider Software sollte ein potentieller Datenlieferant drei Voraussetzungen prüfen: * Die zu veröffentlichenden Sammlungs- oder Beobachtungsdaten müssen in einem SQL-fähigen relationalen Datenbankmanagementsystem vorhanden sein oder in einem Format, zu dem ein Zugang über ODBC (Open Database Connectivity) möglich ist, wie z.B. Microsoft Excel. Nicht-relationale Datenbankmanagementsysteme werden nicht unterstützt. Derzeit kann sich die Provider Software mit den folgenden Datenbanksystemen verbinden: Microsoft Access, Microsoft SQL Server, MySQL, PostgreSQL, Oracle, Foxpro, Sybase, 4D, DB2, Firebird, Microsoft Excel. * Zur Installation der Provider Software muss ein dauerhaft mit dem Internet verbundener Webserver vorhanden sein (Apache oder Microsoft IIS). Dies kann ein Server des Datenproviders oder eines kommerziellen Hosting-Dienstes sein. Eine Veröffentlichung über eine Dial-up-Verbindung ist nicht möglich. * Auf diesem Webserver muss die Installation von Python (insofern es noch nicht installiert ist, was z.B. bei vielen gängigen Linux-Installationen der Fall ist) sowie zusätzlicher Python-Pakete möglich sein. Dies ist für Mitarbeiter der IT üblicherweise keine Hürde, solange dem keine institutionellen Richtlinien entgegenstehen. Für die optionale Erzeugung von DarwinCore-Archiven ist zusätzlich eine Java Runtime Environment nötig. ==Vorgehensweise== Sollte die Wahl auf die BioCASe Provider Software fallen (bei der Entscheidungsfindung hilft das [[Special:Contact|BioCASe Team]] gerne weiter), umfasst eine Anbindung von Daten üblicherweise die folgenden Schritte: # Entscheiden, welche Daten an das Netzwerk angebunden werden sollen. Dazu gehören die Klärung der Frage, wem die Daten eigentlich gehören, sowie die Festlegung der Nutzungsbedingungen/-beschränkungen und einer Lizenz. Darüber hinaus sollte geprüft werden, welche Daten '''nicht''' veröffentlicht werden sollen, weil sie gefährdete Arten betreffen. In diesem Fall können bestimmte Datensätze komplett von der Veröffentlichung ausgeschlossen werden oder bestimmte Teile davon – z.B. die Koordinaten des Fundortes – gezielt verunschärft werden. # Vorbereitung der Datenbank auf die Veröffentlichung. Dabei müssen die Entscheidung getroffen werden, ob die Live-Datenbank oder ein regelmäßig erzeugter Snapshot angebunden werden soll, und die Tabelle für die Metadaten angelegt und befüllt werden. In einigen Fällen sind weitere Datentransformationen nötig, siehe [[Preparation|Tutorial]]. # Installation der Provider Software, siehe [[Installation|Tutorial]]. # Anlegen einer BioCASe Datenquelle, Verbinden mit der Datenbank und Definition der Datenbankstruktur, siehe [[DatasourceSetup|Tutorial]]. # Anlegen eines [[ABCD2Mapping|ABCD2 Mappings]]. # Testen und Debuggen des BioCASe Webservices, siehe [[Debugging|Tutorial]]. # Nach Abschluss der Konfiguration kann der neue BioCASe-Webservice im Wunsch-Datennetzwerk angemeldet werden, z.B. bei [http://www.gbif.org GBIF]. Wir empfehlen jedoch, davor kurz das [[Special:Contact|BioCASe-Team zu kontaktieren]], das einen Blick auf den neuen Webservice werfen und auf Richtigkeit prüfen kann. Falsche Mappings lassen sich umso leichter korrigieren, je früher sie entdeckt werden. ==Support== Bei Problemen während Installation und Konfiguration empfehlen wir * In den [[FAQ]] nachzuschlagen, ob das Problem und eine Lösung dort beschrieben sind, * Befreundete Nutzer der BioCASe Provider Software um Rat zu fragen, oder * Kontakt mit dem [[Special:Contact|BioCASe Team]] aufzunehmen. Bitte dabei genau das verwendete System (verwendetes BioCASe-Version sowie Betriebs/Datenbankmanagementsystem), Fehlermeldungen und die Schritte, die dazu führten, beschreiben. Screenshots sind dabei immer von Vorteil. b0ceeb2f568cf08bbdbd7417dd5c6f0d26418edd 826 825 2015-01-19T15:02:21Z JoergHoletschek 3 wikitext text/x-wiki ==Was ist die BioCASe Provider Software?== Die BioCASe Provider Software ist eine Middleware, also eine Software, die den Datenaustausch zwischen zwei voneinander entfernten Systemen erlaubt. Sie ermöglicht, Daten aus einem relationalen Datenbanksystem (z.B. einem Sammlungsverwaltungssystem) in ein Datennetzwerk einzuspeisen. Nach der Installation und Konfiguration von BioCASe sind die Daten als BioCASe Webservice im Internet verfügbar und können mittels BioCASe-Protokollanfragen abgerufen werden. Die Provider Software ist nicht abhängig von einem bestimmten Datenstandard und lässt sich mit einem beliebigen Datenschema verwenden. Die Provider Software muss auf einem Webserver des Datenlieferanten installiert werden. Kern ist der sogenannte PyWrapper, eine in Python entwickelte XML/CGI-Datenschnittstelle, die einen standardisierten Zugriff auf eine Vielzahl von Datenbankmanagementsystemen mit beliebigen Datenmodellen ermöglicht. Um diese Schnittstelle herum existieren verschiedene Tools, die die Einrichtung, Konfiguration und den Test der BioCASe-Installation und –Webservices vereinfachen. Obwohl BioCASe prinzipiell mit jedem XML-Datenschema verwendet werden kann, ist das Haupteinsatzfeld die Anbindung von biologischen Beleg- und Beobachtungsdaten an Primärdatennetzwerke zur Artenvielfalt wie dem [http://www.biocase.org BioCASe Netzwerk] und der [http://www.gbif.org Global Biodiversity Information Facility]. ==Voraussetzungen für den Einsatz der BioCASe Provider Software== Vor dem Einsatz der Provider Software sollte ein potentieller Datenlieferant drei Voraussetzungen prüfen: * Die zu veröffentlichenden Sammlungs- oder Beobachtungsdaten müssen in einem SQL-fähigen relationalen Datenbankmanagementsystem vorhanden sein oder in einem Format, zu dem ein Zugang über ODBC (Open Database Connectivity) möglich ist, wie z.B. Microsoft Excel. Nicht-relationale Datenbankmanagementsysteme werden nicht unterstützt. Derzeit kann sich die Provider Software mit den folgenden Datenbanksystemen verbinden: Microsoft Access, Microsoft SQL Server, MySQL, PostgreSQL, Oracle, Foxpro, Sybase, 4D, DB2, Firebird, Microsoft Excel. * Zur Installation der Provider Software muss ein dauerhaft mit dem Internet verbundener Webserver vorhanden sein (Apache oder Microsoft IIS). Dies kann ein Server des Datenproviders oder eines kommerziellen Hosting-Dienstes sein. Eine Veröffentlichung über eine Dial-up-Verbindung ist nicht möglich. * Auf diesem Webserver muss die Installation von Python (insofern es noch nicht installiert ist, was z.B. bei vielen gängigen Linux-Installationen der Fall ist) sowie zusätzlicher Python-Pakete möglich sein. Dies ist für Mitarbeiter der IT üblicherweise keine Hürde, solange dem keine institutionellen Richtlinien entgegenstehen. Für die optionale Erzeugung von DarwinCore-Archiven ist zusätzlich eine Java Runtime Environment nötig. ==Vorgehensweise== Sollte die Wahl auf die BioCASe Provider Software fallen (bei der Entscheidungsfindung hilft das [[Special:Contact|BioCASe Team]] gerne weiter), umfasst eine Anbindung von Daten üblicherweise die folgenden Schritte: # Entscheiden, welche Daten an das Netzwerk angebunden werden sollen. Dazu gehören die Klärung der Frage, wem die Daten eigentlich gehören, sowie die Festlegung der Nutzungsbedingungen/-beschränkungen und einer Lizenz. Darüber hinaus sollte geprüft werden, welche Daten '''nicht''' veröffentlicht werden sollen, weil sie gefährdete Arten betreffen. In diesem Fall können bestimmte Datensätze komplett von der Veröffentlichung ausgeschlossen werden oder bestimmte Teile davon – z.B. die Koordinaten des Fundortes – gezielt verunschärft werden. # Vorbereitung der Datenbank auf die Veröffentlichung. Dabei müssen die Entscheidung getroffen werden, ob die Live-Datenbank oder ein regelmäßig erzeugter Snapshot angebunden werden soll, und die Tabelle für die Metadaten angelegt und befüllt werden. In einigen Fällen sind weitere Datentransformationen nötig, siehe [[Preparation|Tutorial]]. # Installation der Provider Software, siehe [[Installation|Tutorial]]. # Anlegen einer BioCASe Datenquelle, Verbinden mit der Datenbank und Definition der Datenbankstruktur, siehe [[DatasourceSetup|Tutorial]]. # Anlegen eines [[ABCD2Mapping|ABCD2 Mappings]]. # Testen und Debuggen des BioCASe Webservices, siehe [[Debugging|Tutorial]]. # Nach Abschluss der Konfiguration kann der neue BioCASe-Webservice im Wunsch-Datennetzwerk angemeldet werden, z.B. bei [http://www.gbif.org GBIF]. Wir empfehlen jedoch, davor kurz das [[Special:Contact|BioCASe-Team zu kontaktieren]], das einen Blick auf den neuen Webservice werfen und auf Richtigkeit prüfen kann. Falsche Mappings lassen sich umso leichter korrigieren, je früher sie entdeckt werden. ==Support== Bei Problemen während Installation und Konfiguration empfehlen wir * In den [[FAQ]] nachzuschlagen, ob das Problem und eine Lösung dort beschrieben sind, * Befreundete Nutzer der BioCASe Provider Software um Rat zu fragen, oder * Kontakt mit dem [[Special:Contact|BioCASe Team]] aufzunehmen. Bitte dabei genau das verwendete System (verwendetes BioCASe-Version sowie Betriebs/Datenbankmanagementsystem), Fehlermeldungen und die Schritte, die dazu führten, beschreiben. Screenshots sind dabei immer von Vorteil. 4f07c2a4927e0a31abdeea1d9469e46778a408ae 828 826 2015-01-19T15:05:43Z JoergHoletschek 3 wikitext text/x-wiki [[http://wiki.bgbm.org/bps/index.php/BeginnersGuide This page in English]] ==Was ist die BioCASe Provider Software?== Die BioCASe Provider Software ist eine Middleware, also eine Software, die den Datenaustausch zwischen zwei voneinander entfernten Systemen erlaubt. Sie ermöglicht, Daten aus einem relationalen Datenbanksystem (z.B. einem Sammlungsverwaltungssystem) in ein Datennetzwerk einzuspeisen. Nach der Installation und Konfiguration von BioCASe sind die Daten als BioCASe Webservice im Internet verfügbar und können mittels BioCASe-Protokollanfragen abgerufen werden. Die Provider Software ist nicht abhängig von einem bestimmten Datenstandard und lässt sich mit einem beliebigen Datenschema verwenden. Die Provider Software muss auf einem Webserver des Datenlieferanten installiert werden. Kern ist der sogenannte PyWrapper, eine in Python entwickelte XML/CGI-Datenschnittstelle, die einen standardisierten Zugriff auf eine Vielzahl von Datenbankmanagementsystemen mit beliebigen Datenmodellen ermöglicht. Um diese Schnittstelle herum existieren verschiedene Tools, die die Einrichtung, Konfiguration und den Test der BioCASe-Installation und –Webservices vereinfachen. Obwohl BioCASe prinzipiell mit jedem XML-Datenschema verwendet werden kann, ist das Haupteinsatzfeld die Anbindung von biologischen Beleg- und Beobachtungsdaten an Primärdatennetzwerke zur Artenvielfalt wie dem [http://www.biocase.org BioCASe Netzwerk] und der [http://www.gbif.org Global Biodiversity Information Facility]. ==Voraussetzungen für den Einsatz der BioCASe Provider Software== Vor dem Einsatz der Provider Software sollte ein potentieller Datenlieferant drei Voraussetzungen prüfen: * Die zu veröffentlichenden Sammlungs- oder Beobachtungsdaten müssen in einem SQL-fähigen relationalen Datenbankmanagementsystem vorhanden sein oder in einem Format, zu dem ein Zugang über ODBC (Open Database Connectivity) möglich ist, wie z.B. Microsoft Excel. Nicht-relationale Datenbankmanagementsysteme werden nicht unterstützt. Derzeit kann sich die Provider Software mit den folgenden Datenbanksystemen verbinden: Microsoft Access, Microsoft SQL Server, MySQL, PostgreSQL, Oracle, Foxpro, Sybase, 4D, DB2, Firebird, Microsoft Excel. * Zur Installation der Provider Software muss ein dauerhaft mit dem Internet verbundener Webserver vorhanden sein (Apache oder Microsoft IIS). Dies kann ein Server des Datenproviders oder eines kommerziellen Hosting-Dienstes sein. Eine Veröffentlichung über eine Dial-up-Verbindung ist nicht möglich. * Auf diesem Webserver muss die Installation von Python (insofern es noch nicht installiert ist, was z.B. bei vielen gängigen Linux-Installationen der Fall ist) sowie zusätzlicher Python-Pakete möglich sein. Dies ist für Mitarbeiter der IT üblicherweise keine Hürde, solange dem keine institutionellen Richtlinien entgegenstehen. Für die optionale Erzeugung von DarwinCore-Archiven ist zusätzlich eine Java Runtime Environment nötig. ==Vorgehensweise== Sollte die Wahl auf die BioCASe Provider Software fallen (bei der Entscheidungsfindung hilft das [[Special:Contact|BioCASe Team]] gerne weiter), umfasst eine Anbindung von Daten üblicherweise die folgenden Schritte: # Entscheiden, welche Daten an das Netzwerk angebunden werden sollen. Dazu gehören die Klärung der Frage, wem die Daten eigentlich gehören, sowie die Festlegung der Nutzungsbedingungen/-beschränkungen und einer Lizenz. Darüber hinaus sollte geprüft werden, welche Daten '''nicht''' veröffentlicht werden sollen, weil sie gefährdete Arten betreffen. In diesem Fall können bestimmte Datensätze komplett von der Veröffentlichung ausgeschlossen werden oder bestimmte Teile davon – z.B. die Koordinaten des Fundortes – gezielt verunschärft werden. # Vorbereitung der Datenbank auf die Veröffentlichung. Dabei müssen die Entscheidung getroffen werden, ob die Live-Datenbank oder ein regelmäßig erzeugter Snapshot angebunden werden soll, und die Tabelle für die Metadaten angelegt und befüllt werden. In einigen Fällen sind weitere Datentransformationen nötig, siehe [[Preparation|Tutorial]]. # Installation der Provider Software, siehe [[Installation|Tutorial]]. # Anlegen einer BioCASe Datenquelle, Verbinden mit der Datenbank und Definition der Datenbankstruktur, siehe [[DatasourceSetup|Tutorial]]. # Anlegen eines [[ABCD2Mapping|ABCD2 Mappings]]. # Testen und Debuggen des BioCASe Webservices, siehe [[Debugging|Tutorial]]. # Nach Abschluss der Konfiguration kann der neue BioCASe-Webservice im Wunsch-Datennetzwerk angemeldet werden, z.B. bei [http://www.gbif.org GBIF]. Wir empfehlen jedoch, davor kurz das [[Special:Contact|BioCASe-Team zu kontaktieren]], das einen Blick auf den neuen Webservice werfen und auf Richtigkeit prüfen kann. Falsche Mappings lassen sich umso leichter korrigieren, je früher sie entdeckt werden. ==Support== Bei Problemen während Installation und Konfiguration empfehlen wir * In den [[FAQ]] nachzuschlagen, ob das Problem und eine Lösung dort beschrieben sind, * Befreundete Nutzer der BioCASe Provider Software um Rat zu fragen, oder * Kontakt mit dem [[Special:Contact|BioCASe Team]] aufzunehmen. Bitte dabei genau das verwendete System (verwendetes BioCASe-Version sowie Betriebs/Datenbankmanagementsystem), Fehlermeldungen und die Schritte, die dazu führten, beschreiben. Screenshots sind dabei immer von Vorteil. b71a421ced0b0b0ddd04df53d14d63bb855dc9b5 0 49 827 593 2015-01-19T15:04:41Z JoergHoletschek 3 wikitext text/x-wiki [[http://wiki.bgbm.org/bps/index.php/BeginnersGuide_de Diese Seite auf Deutsch]] == What is the BioCASe Provider Software? == The BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. The Provider Software is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. The Provider Software is a suite of tools that need to be installed on the data provider’s web server. The core component is the PyWrapper, an XML/CGI database interface written in Python that allows a standardized access to a variety of database management systems and arbitrarily structured databases. Grouped around this are a number of tools for configuring, testing and debugging a BioCASe installation or web service. Even though BioCASe can be used for any conceptual XML data schema, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the [http://www.biocase.org BioCASe network] and the [http://www.gbif.org Global Biodiversity Information Facility]. == Requirements for Using the BPS == Before using the BioCASe Provider Software, three requirements need to be checked: * For one thing, the data to be published needs to be stored in an SQL compliant relational database management system (DBMS), with the data model being well documented or at least knowledge about the data model existing on someone’s head. Non-SQL DBMS are not supported. Currently, the BioCASe Provider Software can connect to the following databases: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * For installing BioCASe, you need to have a web server (Apache or Microsoft IIS) that is permanently connected to the Internet. This can be an institutional server or a hosting service (even though it’s usually a pain to get privileges to install software for the latter). Dial-up connections won’t work. * For the web server in question, you must have the possibility to install Python (if it is not installed yet) and Python packages. Even though this can be done by the IT staff, you should make sure that there are no institutional policies colliding. If you want to create DarwinCore Archives, a Java Runtime Environment is required. == Steps == Once you’ve made sure the BioCASe Provider Software is the right choice for you ([[Special:Contact|contact the BioCASe team]] if you feel unsafe with your decision), you need to do the following steps: # Think about which information you want to publish to the network. This involves settling the question of ownership of the data and the terms of use, license and usage restrictions for the published data. Also you should consider which information should '''not''' be published because they might affect endangered species (options are to block certain occurrences from publication or to blur the exact locality information). # Prepare the database for publication as described in the [[Preparation]] tutorial. This includes the decision on whether to publish the live database or a snapshot, the creation of a table for metadata and, in certain cases, some data transformation. # Follow the [[Installation]] Tutorial to install the BioCASe Provider Software. # Set up a BioCASe Datasource and connect it to your database, then configure the database structure as described in the [[DatasourceSetup]] tutorial. # Follow the [[ABCD2Mapping]] tutorial to create a mapping for the schema you want to use. # Test and debug your BioCASe web service. Read the [[Debugging]] tutorial to learn how to do this. # Once you’re done with setting up and configuring the BioCASe Provider Software, you can register the BioCASe web service with the information network you want to publish your data to, for example with [http://www.gbif.org GBIF]. We strongly recommend to [[Special:Contact|contact the BioCASe team]] for checking your web service first as we might be able to foresee problems that are more difficult to resolve after registration. == Support == In case you run into difficulties during the installation and setup, you should * have a look at the [[FAQ]] which might list the solution for your question, * ask other BioCASe data providers for help, * get in contact with the [[Special:Contact|BioCASe team]]. Please describe exactly you environment (operating system, DBMS used), the steps that cause the problem and any error messages and hints you get. Screenshot are a big plus here. 9afadbd2705affd05d35ce949e3b17c5729ecd52 847 827 2015-08-26T16:28:41Z JoergHoletschek 3 /* Requirements for Using the BPS */ wikitext text/x-wiki [[http://wiki.bgbm.org/bps/index.php/BeginnersGuide_de Diese Seite auf Deutsch]] == What is the BioCASe Provider Software? == The BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. The Provider Software is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. The Provider Software is a suite of tools that need to be installed on the data provider’s web server. The core component is the PyWrapper, an XML/CGI database interface written in Python that allows a standardized access to a variety of database management systems and arbitrarily structured databases. Grouped around this are a number of tools for configuring, testing and debugging a BioCASe installation or web service. Even though BioCASe can be used for any conceptual XML data schema, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the [http://www.biocase.org BioCASe network] and the [http://www.gbif.org Global Biodiversity Information Facility]. == Requirements for Using the BPS == Before using the BioCASe Provider Software, four requirements need to be checked: * For one thing, the data to be published needs to be stored in an SQL compliant relational database management system (DBMS), with the data model being well documented or at least knowledge about the data model existing on someone’s head. Non-SQL DBMS are not supported. Currently, the BioCASe Provider Software can connect to the following databases: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * For installing BioCASe, you need to have a web server (Apache or Microsoft IIS) that is permanently connected to the Internet. This can be an institutional server or a hosting service (even though it’s usually a pain to get privileges to install software for the latter). Dial-up connections won’t work. * For the web server in question, you must have the possibility to install Python (if it is not installed yet) and Python packages. Even though this can be done by the IT staff, you should make sure that there are no institutional policies colliding. If you want to create DarwinCore Archives, a Java Runtime Environment is required. * A direct connection can be established from the web server that will run the BPS to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL, 1433 for SQL Server). == Steps == Once you’ve made sure the BioCASe Provider Software is the right choice for you ([[Special:Contact|contact the BioCASe team]] if you feel unsafe with your decision), you need to do the following steps: # Think about which information you want to publish to the network. This involves settling the question of ownership of the data and the terms of use, license and usage restrictions for the published data. Also you should consider which information should '''not''' be published because they might affect endangered species (options are to block certain occurrences from publication or to blur the exact locality information). # Prepare the database for publication as described in the [[Preparation]] tutorial. This includes the decision on whether to publish the live database or a snapshot, the creation of a table for metadata and, in certain cases, some data transformation. # Follow the [[Installation]] Tutorial to install the BioCASe Provider Software. # Set up a BioCASe Datasource and connect it to your database, then configure the database structure as described in the [[DatasourceSetup]] tutorial. # Follow the [[ABCD2Mapping]] tutorial to create a mapping for the schema you want to use. # Test and debug your BioCASe web service. Read the [[Debugging]] tutorial to learn how to do this. # Once you’re done with setting up and configuring the BioCASe Provider Software, you can register the BioCASe web service with the information network you want to publish your data to, for example with [http://www.gbif.org GBIF]. We strongly recommend to [[Special:Contact|contact the BioCASe team]] for checking your web service first as we might be able to foresee problems that are more difficult to resolve after registration. == Support == In case you run into difficulties during the installation and setup, you should * have a look at the [[FAQ]] which might list the solution for your question, * ask other BioCASe data providers for help, * get in contact with the [[Special:Contact|BioCASe team]]. Please describe exactly you environment (operating system, DBMS used), the steps that cause the problem and any error messages and hints you get. Screenshot are a big plus here. 8f5ff4e5e8fc8d12ad0fe95ce64cc1d84e295df1 0 80 834 743 2015-06-25T13:05:12Z JoergHoletschek 3 wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later.<br>If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousand or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The Global Biodiversity Information Facility (GBIF), for example, can use this method for fast indexing of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==Managing Archives== The section <small>Existing Archives</small> allows you download, remove and convert archives or view the log of the archiving process: [[File:xmlArchivesNew.png|border|775px]] The archives are grouped by schema, with DarwinCore archives making up an additional group. You can use the minus or plus symbol next to the group header to expand or collapse all files of that group; for web services publishing dozens of datasets, this might be handy to reduce scrolling on the page. The <small>Log </small> link in the group header will open the log file for the respective archiving process in a separate tab. For each archive file, the number of records stored in this file, the date of creation and the file size are listed. XML archives may have a number printed in the <small>Dropped</small> column, meaning some records have been dropped due to missing mandatory data elements. This doesn't mean the archive is invalid, but you should review the mapping or your source data to find the reasons for these lost records. Clicking the archive name will download the archive file, so it can be saved to disk or opened with an archive viewing tool. Pressing the trash symbol at the very right will delete the archive; if you use the trash symbol in the group header, all archives of this schema/type will be removed. For ABCD 2.06 and HISPID 5 XML archives, you will see an additional <small>DwC</small> link next to the trash symbol. This allows you to selectively convert a single dataset from XML to DarwinCore format. In contrast to using the <small>Create DwC Archive</small> button described below, this will spare the XML archiving of the whole web service and just create a DarwinCore archive for the selected dataset, which will be significantly faster. If you use the link in the group header, all archives of the schema will be converted (again without triggering XML archiving first). ==XML Archives== If you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. The section <small>Create/Update Archives</small> allows you to create new XML archive(s) for each schema mapped. Simply select the schema and press <small>Create XML Archive</small>. As mentioned above, archiving ABCD 2.06 or HISPID 5 is dataset-aware, resulting in one XML archive per dataset published. For all other schemas, a single archive file will be created. The paging size specifies how many records will be bundled into one XML document inside the archive; the default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgressNew.png|border]] You can always cancel the archiving process by pressing the <small>Cancel</small> link in the Status/Messages box. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Existing Archives</small> list, or, if it’s still being processed, as <small>Processing</small> in the Status/Messages box. If the process failed for some reason (or you decided to cancel it), a new box <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create XML archives for one schema at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. DarwinCore archives store only one dataset; so as with ABCD 2.06 and HISPID 5 XML archives, DwC-archiving a web service might result in several DarwinCore archives. DarwinCore archives are not created directly, but from the respective ABCD/HISPID archives. So DwC-archiving a web service will first create ACBD archives, which will be converted into the respective DwC archives subsequently. ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the library test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive before, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the archive name listed under ''Existing Archives''. The XML archiving step and the parameter <small>Paging size</small> are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgressNew.png|border]] You can cancel the transformation process by clicking the <small>Cancel</small> link in the Status/Messages box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from the ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: Conversion finished.'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. This parameter corresponds to the paging size that can be set on the archiving page. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. == Discovery of Archives == 26558cda20f03f0a43ef5afa4a75ec170090c4a2 835 834 2015-06-25T13:16:22Z JoergHoletschek 3 /* Discovery of Archives */ wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later.<br>If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousand or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The Global Biodiversity Information Facility (GBIF), for example, can use this method for fast indexing of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==Managing Archives== The section <small>Existing Archives</small> allows you download, remove and convert archives or view the log of the archiving process: [[File:xmlArchivesNew.png|border|775px]] The archives are grouped by schema, with DarwinCore archives making up an additional group. You can use the minus or plus symbol next to the group header to expand or collapse all files of that group; for web services publishing dozens of datasets, this might be handy to reduce scrolling on the page. The <small>Log </small> link in the group header will open the log file for the respective archiving process in a separate tab. For each archive file, the number of records stored in this file, the date of creation and the file size are listed. XML archives may have a number printed in the <small>Dropped</small> column, meaning some records have been dropped due to missing mandatory data elements. This doesn't mean the archive is invalid, but you should review the mapping or your source data to find the reasons for these lost records. Clicking the archive name will download the archive file, so it can be saved to disk or opened with an archive viewing tool. Pressing the trash symbol at the very right will delete the archive; if you use the trash symbol in the group header, all archives of this schema/type will be removed. For ABCD 2.06 and HISPID 5 XML archives, you will see an additional <small>DwC</small> link next to the trash symbol. This allows you to selectively convert a single dataset from XML to DarwinCore format. In contrast to using the <small>Create DwC Archive</small> button described below, this will spare the XML archiving of the whole web service and just create a DarwinCore archive for the selected dataset, which will be significantly faster. If you use the link in the group header, all archives of the schema will be converted (again without triggering XML archiving first). ==XML Archives== If you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. The section <small>Create/Update Archives</small> allows you to create new XML archive(s) for each schema mapped. Simply select the schema and press <small>Create XML Archive</small>. As mentioned above, archiving ABCD 2.06 or HISPID 5 is dataset-aware, resulting in one XML archive per dataset published. For all other schemas, a single archive file will be created. The paging size specifies how many records will be bundled into one XML document inside the archive; the default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgressNew.png|border]] You can always cancel the archiving process by pressing the <small>Cancel</small> link in the Status/Messages box. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Existing Archives</small> list, or, if it’s still being processed, as <small>Processing</small> in the Status/Messages box. If the process failed for some reason (or you decided to cancel it), a new box <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create XML archives for one schema at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. DarwinCore archives store only one dataset; so as with ABCD 2.06 and HISPID 5 XML archives, DwC-archiving a web service might result in several DarwinCore archives. DarwinCore archives are not created directly, but from the respective ABCD/HISPID archives. So DwC-archiving a web service will first create ACBD archives, which will be converted into the respective DwC archives subsequently. ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the library test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive before, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the archive name listed under ''Existing Archives''. The XML archiving step and the parameter <small>Paging size</small> are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgressNew.png|border]] You can cancel the transformation process by clicking the <small>Cancel</small> link in the Status/Messages box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from the ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: Conversion finished.'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. This parameter corresponds to the paging size that can be set on the archiving page. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. == Discovery of Archives == Archives available for a given data source can be discovered through the so-called dataset inventory. To get this, just append ''&inventory=1'' to the access URL of the data source, for example http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=BoBO&inventory=1. The dataset will list all datasets published by a given data source, each with its title and GUID. If archives exist for this dataset, they will be listed in ''archives''. For XML archives, the attribute namespace will store the namespace of the schema, for DarwinCore archives there will an attribute ''rowType" with the value '"http://rs.tdwg.org/dwc/terms/Occurrence". 027e9cc495bbebe63fd2cc9ae398f020595193d7 836 835 2015-06-25T13:17:52Z JoergHoletschek 3 /* Discovery of Archives */ wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later.<br>If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousand or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The Global Biodiversity Information Facility (GBIF), for example, can use this method for fast indexing of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==Managing Archives== The section <small>Existing Archives</small> allows you download, remove and convert archives or view the log of the archiving process: [[File:xmlArchivesNew.png|border|775px]] The archives are grouped by schema, with DarwinCore archives making up an additional group. You can use the minus or plus symbol next to the group header to expand or collapse all files of that group; for web services publishing dozens of datasets, this might be handy to reduce scrolling on the page. The <small>Log </small> link in the group header will open the log file for the respective archiving process in a separate tab. For each archive file, the number of records stored in this file, the date of creation and the file size are listed. XML archives may have a number printed in the <small>Dropped</small> column, meaning some records have been dropped due to missing mandatory data elements. This doesn't mean the archive is invalid, but you should review the mapping or your source data to find the reasons for these lost records. Clicking the archive name will download the archive file, so it can be saved to disk or opened with an archive viewing tool. Pressing the trash symbol at the very right will delete the archive; if you use the trash symbol in the group header, all archives of this schema/type will be removed. For ABCD 2.06 and HISPID 5 XML archives, you will see an additional <small>DwC</small> link next to the trash symbol. This allows you to selectively convert a single dataset from XML to DarwinCore format. In contrast to using the <small>Create DwC Archive</small> button described below, this will spare the XML archiving of the whole web service and just create a DarwinCore archive for the selected dataset, which will be significantly faster. If you use the link in the group header, all archives of the schema will be converted (again without triggering XML archiving first). ==XML Archives== If you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. The section <small>Create/Update Archives</small> allows you to create new XML archive(s) for each schema mapped. Simply select the schema and press <small>Create XML Archive</small>. As mentioned above, archiving ABCD 2.06 or HISPID 5 is dataset-aware, resulting in one XML archive per dataset published. For all other schemas, a single archive file will be created. The paging size specifies how many records will be bundled into one XML document inside the archive; the default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgressNew.png|border]] You can always cancel the archiving process by pressing the <small>Cancel</small> link in the Status/Messages box. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Existing Archives</small> list, or, if it’s still being processed, as <small>Processing</small> in the Status/Messages box. If the process failed for some reason (or you decided to cancel it), a new box <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create XML archives for one schema at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. DarwinCore archives store only one dataset; so as with ABCD 2.06 and HISPID 5 XML archives, DwC-archiving a web service might result in several DarwinCore archives. DarwinCore archives are not created directly, but from the respective ABCD/HISPID archives. So DwC-archiving a web service will first create ACBD archives, which will be converted into the respective DwC archives subsequently. ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the library test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive before, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the archive name listed under ''Existing Archives''. The XML archiving step and the parameter <small>Paging size</small> are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgressNew.png|border]] You can cancel the transformation process by clicking the <small>Cancel</small> link in the Status/Messages box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from the ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: Conversion finished.'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. This parameter corresponds to the paging size that can be set on the archiving page. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. == Discovery of Archives == Archives available for a given data source can be discovered through the so-called dataset inventory. To get this, just append ''&inventory=1'' to the access URL of the data source, for example [http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=BoBO&inventory=1]. The dataset will list all datasets published by a given data source, each with its title and GUID. If archives exist for this dataset, they will be listed in ''archives''. For XML archives, the attribute namespace will store the namespace of the schema, for DarwinCore archives there will an attribute ''rowType" with the value '"http://rs.tdwg.org/dwc/terms/Occurrence". bec79d35dac608fd77c16ed6d3586bfe2b8a5893 837 836 2015-06-25T13:18:36Z JoergHoletschek 3 /* Discovery of Archives */ wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later.<br>If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousand or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The Global Biodiversity Information Facility (GBIF), for example, can use this method for fast indexing of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==Managing Archives== The section <small>Existing Archives</small> allows you download, remove and convert archives or view the log of the archiving process: [[File:xmlArchivesNew.png|border|775px]] The archives are grouped by schema, with DarwinCore archives making up an additional group. You can use the minus or plus symbol next to the group header to expand or collapse all files of that group; for web services publishing dozens of datasets, this might be handy to reduce scrolling on the page. The <small>Log </small> link in the group header will open the log file for the respective archiving process in a separate tab. For each archive file, the number of records stored in this file, the date of creation and the file size are listed. XML archives may have a number printed in the <small>Dropped</small> column, meaning some records have been dropped due to missing mandatory data elements. This doesn't mean the archive is invalid, but you should review the mapping or your source data to find the reasons for these lost records. Clicking the archive name will download the archive file, so it can be saved to disk or opened with an archive viewing tool. Pressing the trash symbol at the very right will delete the archive; if you use the trash symbol in the group header, all archives of this schema/type will be removed. For ABCD 2.06 and HISPID 5 XML archives, you will see an additional <small>DwC</small> link next to the trash symbol. This allows you to selectively convert a single dataset from XML to DarwinCore format. In contrast to using the <small>Create DwC Archive</small> button described below, this will spare the XML archiving of the whole web service and just create a DarwinCore archive for the selected dataset, which will be significantly faster. If you use the link in the group header, all archives of the schema will be converted (again without triggering XML archiving first). ==XML Archives== If you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. The section <small>Create/Update Archives</small> allows you to create new XML archive(s) for each schema mapped. Simply select the schema and press <small>Create XML Archive</small>. As mentioned above, archiving ABCD 2.06 or HISPID 5 is dataset-aware, resulting in one XML archive per dataset published. For all other schemas, a single archive file will be created. The paging size specifies how many records will be bundled into one XML document inside the archive; the default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgressNew.png|border]] You can always cancel the archiving process by pressing the <small>Cancel</small> link in the Status/Messages box. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Existing Archives</small> list, or, if it’s still being processed, as <small>Processing</small> in the Status/Messages box. If the process failed for some reason (or you decided to cancel it), a new box <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create XML archives for one schema at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. DarwinCore archives store only one dataset; so as with ABCD 2.06 and HISPID 5 XML archives, DwC-archiving a web service might result in several DarwinCore archives. DarwinCore archives are not created directly, but from the respective ABCD/HISPID archives. So DwC-archiving a web service will first create ACBD archives, which will be converted into the respective DwC archives subsequently. ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the library test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive before, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the archive name listed under ''Existing Archives''. The XML archiving step and the parameter <small>Paging size</small> are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgressNew.png|border]] You can cancel the transformation process by clicking the <small>Cancel</small> link in the Status/Messages box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from the ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: Conversion finished.'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. This parameter corresponds to the paging size that can be set on the archiving page. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. == Discovery of Archives == Archives available for a given data source can be discovered through the so-called dataset inventory. To get this, just append ''&inventory=1'' to the access URL of the data source, for example http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=BoBO&inventory=1. The dataset will list all datasets published by a given data source, each with its title and GUID. If archives exist for this dataset, they will be listed in ''archives''. For XML archives, the attribute namespace will store the namespace of the schema, for DarwinCore archives there will an attribute ''rowType" with the value '"http://rs.tdwg.org/dwc/terms/Occurrence". 027e9cc495bbebe63fd2cc9ae398f020595193d7 838 837 2015-06-25T13:22:49Z JoergHoletschek 3 /* Discovery of Archives */ wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later.<br>If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousand or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The Global Biodiversity Information Facility (GBIF), for example, can use this method for fast indexing of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==Managing Archives== The section <small>Existing Archives</small> allows you download, remove and convert archives or view the log of the archiving process: [[File:xmlArchivesNew.png|border|775px]] The archives are grouped by schema, with DarwinCore archives making up an additional group. You can use the minus or plus symbol next to the group header to expand or collapse all files of that group; for web services publishing dozens of datasets, this might be handy to reduce scrolling on the page. The <small>Log </small> link in the group header will open the log file for the respective archiving process in a separate tab. For each archive file, the number of records stored in this file, the date of creation and the file size are listed. XML archives may have a number printed in the <small>Dropped</small> column, meaning some records have been dropped due to missing mandatory data elements. This doesn't mean the archive is invalid, but you should review the mapping or your source data to find the reasons for these lost records. Clicking the archive name will download the archive file, so it can be saved to disk or opened with an archive viewing tool. Pressing the trash symbol at the very right will delete the archive; if you use the trash symbol in the group header, all archives of this schema/type will be removed. For ABCD 2.06 and HISPID 5 XML archives, you will see an additional <small>DwC</small> link next to the trash symbol. This allows you to selectively convert a single dataset from XML to DarwinCore format. In contrast to using the <small>Create DwC Archive</small> button described below, this will spare the XML archiving of the whole web service and just create a DarwinCore archive for the selected dataset, which will be significantly faster. If you use the link in the group header, all archives of the schema will be converted (again without triggering XML archiving first). ==XML Archives== If you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. The section <small>Create/Update Archives</small> allows you to create new XML archive(s) for each schema mapped. Simply select the schema and press <small>Create XML Archive</small>. As mentioned above, archiving ABCD 2.06 or HISPID 5 is dataset-aware, resulting in one XML archive per dataset published. For all other schemas, a single archive file will be created. The paging size specifies how many records will be bundled into one XML document inside the archive; the default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgressNew.png|border]] You can always cancel the archiving process by pressing the <small>Cancel</small> link in the Status/Messages box. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Existing Archives</small> list, or, if it’s still being processed, as <small>Processing</small> in the Status/Messages box. If the process failed for some reason (or you decided to cancel it), a new box <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create XML archives for one schema at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. DarwinCore archives store only one dataset; so as with ABCD 2.06 and HISPID 5 XML archives, DwC-archiving a web service might result in several DarwinCore archives. DarwinCore archives are not created directly, but from the respective ABCD/HISPID archives. So DwC-archiving a web service will first create ACBD archives, which will be converted into the respective DwC archives subsequently. ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the library test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive before, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the archive name listed under ''Existing Archives''. The XML archiving step and the parameter <small>Paging size</small> are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgressNew.png|border]] You can cancel the transformation process by clicking the <small>Cancel</small> link in the Status/Messages box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from the ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: Conversion finished.'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. This parameter corresponds to the paging size that can be set on the archiving page. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. == Discovery of Archives == Archives available for a given data source can be discovered through the so-called dataset inventory. To get this, just append ''&inventory=1'' to the access URL of the data source, for example http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=BoBO&inventory=1. <syntaxhighlight> <dsi:inventory xsi:schemaLocation="http://www.biocase.org/schemas/dsi/1.0 http://www.bgbm.org/biodivinf/schema/dsi_1_0.xsd"> <!--XML generated by BioCASE PyWrapper software version 3.5.3. Made in Berlin.--> <dsi:status>OK</dsi:status> <dsi:created>2015-06-25T15:22:51.805000</dsi:created> <dsi:service_url>http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=BoBO</dsi:service_url> <dsi:datasets><dsi:dataset> <dsi:title>BoBO - Botanic Garden and Botanical Museum Berlin-Dahlem Observations</dsi:title> <dsi:id>BoBO - Botanic Garden and Botanical Museum Berlin-Dahlem Observations</dsi:id> <dsi:archives> <dsi:archive filesize="143137" modified="2015-05-06T16:06:19.346807" namespace="http://www.tdwg.org/schemas/abcd/2.06" rcount="823">http://ww3.bgbm.org/biocase/downloads/BoBO/BoBO%20-%20Botanic%20Garden%20and%20Botanical%20Museum%20Berlin-Dahlem%20Observations.ABCD_2.06.zip</dsi:archive> <dsi:archive filesize="128502" modified="2015-05-06T16:06:34.768532" rcount="823" rowType="http://rs.tdwg.org/dwc/terms/Occurrence">http://ww3.bgbm.org/biocase/downloads/BoBO/BoBO%20-%20Botanic%20Garden%20and%20Botanical%20Museum%20Berlin-Dahlem%20Observations.DwCA.zip</dsi:archive> </dsi:archives> </dsi:dataset> </dsi:datasets> <dsi:diagnostics> ... </dsi:diagnostics> </dsi:inventory> </syntaxhighlight> The dataset will list all datasets published by a given data source, each with its title and GUID. If archives exist for this dataset, they will be listed in ''archives''. For XML archives, the attribute namespace will store the namespace of the schema, for DarwinCore archives there will an attribute ''rowType" with the value '"http://rs.tdwg.org/dwc/terms/Occurrence". 1ec186cbea8b9aa4194d549e92be35900110b40d 839 838 2015-06-25T13:34:23Z JoergHoletschek 3 /* Discovery of Archives */ wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later.<br>If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousand or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The Global Biodiversity Information Facility (GBIF), for example, can use this method for fast indexing of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==Managing Archives== The section <small>Existing Archives</small> allows you download, remove and convert archives or view the log of the archiving process: [[File:xmlArchivesNew.png|border|775px]] The archives are grouped by schema, with DarwinCore archives making up an additional group. You can use the minus or plus symbol next to the group header to expand or collapse all files of that group; for web services publishing dozens of datasets, this might be handy to reduce scrolling on the page. The <small>Log </small> link in the group header will open the log file for the respective archiving process in a separate tab. For each archive file, the number of records stored in this file, the date of creation and the file size are listed. XML archives may have a number printed in the <small>Dropped</small> column, meaning some records have been dropped due to missing mandatory data elements. This doesn't mean the archive is invalid, but you should review the mapping or your source data to find the reasons for these lost records. Clicking the archive name will download the archive file, so it can be saved to disk or opened with an archive viewing tool. Pressing the trash symbol at the very right will delete the archive; if you use the trash symbol in the group header, all archives of this schema/type will be removed. For ABCD 2.06 and HISPID 5 XML archives, you will see an additional <small>DwC</small> link next to the trash symbol. This allows you to selectively convert a single dataset from XML to DarwinCore format. In contrast to using the <small>Create DwC Archive</small> button described below, this will spare the XML archiving of the whole web service and just create a DarwinCore archive for the selected dataset, which will be significantly faster. If you use the link in the group header, all archives of the schema will be converted (again without triggering XML archiving first). ==XML Archives== If you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. The section <small>Create/Update Archives</small> allows you to create new XML archive(s) for each schema mapped. Simply select the schema and press <small>Create XML Archive</small>. As mentioned above, archiving ABCD 2.06 or HISPID 5 is dataset-aware, resulting in one XML archive per dataset published. For all other schemas, a single archive file will be created. The paging size specifies how many records will be bundled into one XML document inside the archive; the default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgressNew.png|border]] You can always cancel the archiving process by pressing the <small>Cancel</small> link in the Status/Messages box. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Existing Archives</small> list, or, if it’s still being processed, as <small>Processing</small> in the Status/Messages box. If the process failed for some reason (or you decided to cancel it), a new box <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create XML archives for one schema at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. DarwinCore archives store only one dataset; so as with ABCD 2.06 and HISPID 5 XML archives, DwC-archiving a web service might result in several DarwinCore archives. DarwinCore archives are not created directly, but from the respective ABCD/HISPID archives. So DwC-archiving a web service will first create ACBD archives, which will be converted into the respective DwC archives subsequently. ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the library test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive before, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the archive name listed under ''Existing Archives''. The XML archiving step and the parameter <small>Paging size</small> are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgressNew.png|border]] You can cancel the transformation process by clicking the <small>Cancel</small> link in the Status/Messages box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from the ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: Conversion finished.'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. This parameter corresponds to the paging size that can be set on the archiving page. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. == Discovery of Archives == Archives available for a given data source can be discovered through the so-called dataset inventory. To get this, just append ''&inventory=1'' to the access URL of the data source, for example http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=BoBO&inventory=1. <syntaxhighlight> <dsi:inventory xsi:schemaLocation="http://www.biocase.org/schemas/dsi/1.0 http://www.bgbm.org/biodivinf/schema/dsi_1_0.xsd"> <!--XML generated by BioCASE PyWrapper software version 3.5.3. Made in Berlin.--> <dsi:status>OK</dsi:status> <dsi:created>2015-06-25T15:22:51.805000</dsi:created> <dsi:service_url>http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=BoBO</dsi:service_url> <dsi:datasets> <dsi:dataset> <dsi:title>BoBO - Botanic Garden and Botanical Museum Berlin-Dahlem Observations</dsi:title> <dsi:id>BoBO - Botanic Garden and Botanical Museum Berlin-Dahlem Observations</dsi:id> <dsi:archives> <dsi:archive filesize="143137" modified="2015-05-06T16:06:19.346807" namespace="http://www.tdwg.org/schemas/abcd/2.06" rcount="823">http://ww3.bgbm.org/biocase/downloads/BoBO/BoBO%20-%20Botanic%20Garden%20and%20Botanical%20Museum%20Berlin-Dahlem%20Observations.ABCD_2.06.zip</dsi:archive> <dsi:archive filesize="128502" modified="2015-05-06T16:06:34.768532" rcount="823" rowType="http://rs.tdwg.org/dwc/terms/Occurrence">http://ww3.bgbm.org/biocase/downloads/BoBO/BoBO%20-%20Botanic%20Garden%20and%20Botanical%20Museum%20Berlin-Dahlem%20Observations.DwCA.zip</dsi:archive> </dsi:archives> </dsi:dataset> </dsi:datasets> <dsi:diagnostics> ... </dsi:diagnostics> </dsi:inventory> </syntaxhighlight> The dataset will list all datasets published by a given data source, each with its title and ID. If archives exist for this dataset, they will be listed in ''archives'', each ''archive'' element storing the URL for the respective file. For XML archives, the attribute ''namespace'' will store the namespace of the schema, for DarwinCore archives there will an attribute ''rowType'' with the value <nowiki>http://rs.tdwg.org/dwc/terms/Occurrence</nowiki>. 6218edd21f2edceced979d4f1d2c83b403a200cf 840 839 2015-06-25T14:41:41Z JoergHoletschek 3 /* Discovery of Archives */ wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later.<br>If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousand or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The Global Biodiversity Information Facility (GBIF), for example, can use this method for fast indexing of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==Managing Archives== The section <small>Existing Archives</small> allows you download, remove and convert archives or view the log of the archiving process: [[File:xmlArchivesNew.png|border|775px]] The archives are grouped by schema, with DarwinCore archives making up an additional group. You can use the minus or plus symbol next to the group header to expand or collapse all files of that group; for web services publishing dozens of datasets, this might be handy to reduce scrolling on the page. The <small>Log </small> link in the group header will open the log file for the respective archiving process in a separate tab. For each archive file, the number of records stored in this file, the date of creation and the file size are listed. XML archives may have a number printed in the <small>Dropped</small> column, meaning some records have been dropped due to missing mandatory data elements. This doesn't mean the archive is invalid, but you should review the mapping or your source data to find the reasons for these lost records. Clicking the archive name will download the archive file, so it can be saved to disk or opened with an archive viewing tool. Pressing the trash symbol at the very right will delete the archive; if you use the trash symbol in the group header, all archives of this schema/type will be removed. For ABCD 2.06 and HISPID 5 XML archives, you will see an additional <small>DwC</small> link next to the trash symbol. This allows you to selectively convert a single dataset from XML to DarwinCore format. In contrast to using the <small>Create DwC Archive</small> button described below, this will spare the XML archiving of the whole web service and just create a DarwinCore archive for the selected dataset, which will be significantly faster. If you use the link in the group header, all archives of the schema will be converted (again without triggering XML archiving first). ==XML Archives== If you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. The section <small>Create/Update Archives</small> allows you to create new XML archive(s) for each schema mapped. Simply select the schema and press <small>Create XML Archive</small>. As mentioned above, archiving ABCD 2.06 or HISPID 5 is dataset-aware, resulting in one XML archive per dataset published. For all other schemas, a single archive file will be created. The paging size specifies how many records will be bundled into one XML document inside the archive; the default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgressNew.png|border]] You can always cancel the archiving process by pressing the <small>Cancel</small> link in the Status/Messages box. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Existing Archives</small> list, or, if it’s still being processed, as <small>Processing</small> in the Status/Messages box. If the process failed for some reason (or you decided to cancel it), a new box <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create XML archives for one schema at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. DarwinCore archives store only one dataset; so as with ABCD 2.06 and HISPID 5 XML archives, DwC-archiving a web service might result in several DarwinCore archives. DarwinCore archives are not created directly, but from the respective ABCD/HISPID archives. So DwC-archiving a web service will first create ACBD archives, which will be converted into the respective DwC archives subsequently. ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the library test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive before, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the archive name listed under ''Existing Archives''. The XML archiving step and the parameter <small>Paging size</small> are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgressNew.png|border]] You can cancel the transformation process by clicking the <small>Cancel</small> link in the Status/Messages box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from the ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: Conversion finished.'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. This parameter corresponds to the paging size that can be set on the archiving page. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. == Discovery of Archives == Archives available for a given data source can be discovered through the so-called dataset inventory. To get this, just append ''&inventory=1'' to the access URL of the data source, for example http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=BoBO&inventory=1. <syntaxhighlight> <dsi:inventory xsi:schemaLocation="http://www.biocase.org/schemas/dsi/1.0 http://www.bgbm.org/biodivinf/schema/dsi_1_0.xsd"> <!--XML generated by BioCASE PyWrapper software version 3.5.3. Made in Berlin.--> <dsi:status>OK</dsi:status> <dsi:created>2015-06-25T15:22:51.805000</dsi:created> <dsi:service_url>http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=BoBO</dsi:service_url> <dsi:datasets> <dsi:dataset> <dsi:title>BoBO - Botanic Garden and Botanical Museum Berlin-Dahlem Observations</dsi:title> <dsi:id>BoBO - Botanic Garden and Botanical Museum Berlin-Dahlem Observations</dsi:id> <dsi:archives> <dsi:archive filesize="143137" modified="2015-05-06T16:06:19.346807" namespace="http://www.tdwg.org/schemas/abcd/2.06" rcount="823">http://ww3.bgbm.org/biocase/downloads/BoBO/BoBO%20-%20Botanic%20Garden%20and%20Botanical%20Museum%20Berlin-Dahlem%20Observations.ABCD_2.06.zip</dsi:archive> <dsi:archive filesize="128502" modified="2015-05-06T16:06:34.768532" rcount="823" rowType="http://rs.tdwg.org/dwc/terms/Occurrence">http://ww3.bgbm.org/biocase/downloads/BoBO/BoBO%20-%20Botanic%20Garden%20and%20Botanical%20Museum%20Berlin-Dahlem%20Observations.DwCA.zip</dsi:archive> </dsi:archives> </dsi:dataset> </dsi:datasets> <dsi:diagnostics> ... </dsi:diagnostics> </dsi:inventory> </syntaxhighlight> The dataset inventory will list all datasets published by a given data source, each with its title and ID. If archives exist for this dataset, they will be listed in ''archives'', each ''archive'' element storing the URL for the respective file. For XML archives, the attribute ''namespace'' will store the namespace of the schema, for DarwinCore archives there will an attribute ''rowType'' with the value <nowiki>http://rs.tdwg.org/dwc/terms/Occurrence</nowiki>. You can also view the dataset inventory using the manual QueryForm: Scroll down to the section ''Configuration Info'' and click on the link ''Inventory''. After a second, it will appear in the text box below. e83a006fcde48dd779e9118d85ee1692ff1047f7 841 840 2015-06-26T07:03:03Z JoergHoletschek 3 /* Discovery of Archives */ wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later.<br>If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousand or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The Global Biodiversity Information Facility (GBIF), for example, can use this method for fast indexing of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==Managing Archives== The section <small>Existing Archives</small> allows you download, remove and convert archives or view the log of the archiving process: [[File:xmlArchivesNew.png|border|775px]] The archives are grouped by schema, with DarwinCore archives making up an additional group. You can use the minus or plus symbol next to the group header to expand or collapse all files of that group; for web services publishing dozens of datasets, this might be handy to reduce scrolling on the page. The <small>Log </small> link in the group header will open the log file for the respective archiving process in a separate tab. For each archive file, the number of records stored in this file, the date of creation and the file size are listed. XML archives may have a number printed in the <small>Dropped</small> column, meaning some records have been dropped due to missing mandatory data elements. This doesn't mean the archive is invalid, but you should review the mapping or your source data to find the reasons for these lost records. Clicking the archive name will download the archive file, so it can be saved to disk or opened with an archive viewing tool. Pressing the trash symbol at the very right will delete the archive; if you use the trash symbol in the group header, all archives of this schema/type will be removed. For ABCD 2.06 and HISPID 5 XML archives, you will see an additional <small>DwC</small> link next to the trash symbol. This allows you to selectively convert a single dataset from XML to DarwinCore format. In contrast to using the <small>Create DwC Archive</small> button described below, this will spare the XML archiving of the whole web service and just create a DarwinCore archive for the selected dataset, which will be significantly faster. If you use the link in the group header, all archives of the schema will be converted (again without triggering XML archiving first). ==XML Archives== If you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. The section <small>Create/Update Archives</small> allows you to create new XML archive(s) for each schema mapped. Simply select the schema and press <small>Create XML Archive</small>. As mentioned above, archiving ABCD 2.06 or HISPID 5 is dataset-aware, resulting in one XML archive per dataset published. For all other schemas, a single archive file will be created. The paging size specifies how many records will be bundled into one XML document inside the archive; the default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgressNew.png|border]] You can always cancel the archiving process by pressing the <small>Cancel</small> link in the Status/Messages box. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Existing Archives</small> list, or, if it’s still being processed, as <small>Processing</small> in the Status/Messages box. If the process failed for some reason (or you decided to cancel it), a new box <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create XML archives for one schema at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. DarwinCore archives store only one dataset; so as with ABCD 2.06 and HISPID 5 XML archives, DwC-archiving a web service might result in several DarwinCore archives. DarwinCore archives are not created directly, but from the respective ABCD/HISPID archives. So DwC-archiving a web service will first create ACBD archives, which will be converted into the respective DwC archives subsequently. ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the library test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive before, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the archive name listed under ''Existing Archives''. The XML archiving step and the parameter <small>Paging size</small> are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgressNew.png|border]] You can cancel the transformation process by clicking the <small>Cancel</small> link in the Status/Messages box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from the ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: Conversion finished.'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. This parameter corresponds to the paging size that can be set on the archiving page. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. == Discovery of Archives == Archives available for a given data source can be discovered through the so-called dataset inventory. To get this, just append ''&inventory=1'' to the access URL of the data source, for example http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=BoBO&inventory=1. The dataset inventory response looks roughly like this (diagnostic messages have been removed for brevity): <syntaxhighlight> <dsi:inventory xsi:schemaLocation="http://www.biocase.org/schemas/dsi/1.0 http://www.bgbm.org/biodivinf/schema/dsi_1_0.xsd"> <!--XML generated by BioCASE PyWrapper software version 3.5.3. Made in Berlin.--> <dsi:status>OK</dsi:status> <dsi:created>2015-06-25T15:22:51.805000</dsi:created> <dsi:service_url>http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=BoBO</dsi:service_url> <dsi:datasets> <dsi:dataset> <dsi:title>BoBO - Botanic Garden and Botanical Museum Berlin-Dahlem Observations</dsi:title> <dsi:id>BoBO - Botanic Garden and Botanical Museum Berlin-Dahlem Observations</dsi:id> <dsi:archives> <dsi:archive filesize="143137" modified="2015-05-06T16:06:19.346807" namespace="http://www.tdwg.org/schemas/abcd/2.06" rcount="823">http://ww3.bgbm.org/biocase/downloads/BoBO/BoBO%20-%20Botanic%20Garden%20and%20Botanical%20Museum%20Berlin-Dahlem%20Observations.ABCD_2.06.zip</dsi:archive> <dsi:archive filesize="128502" modified="2015-05-06T16:06:34.768532" rcount="823" rowType="http://rs.tdwg.org/dwc/terms/Occurrence">http://ww3.bgbm.org/biocase/downloads/BoBO/BoBO%20-%20Botanic%20Garden%20and%20Botanical%20Museum%20Berlin-Dahlem%20Observations.DwCA.zip</dsi:archive> </dsi:archives> </dsi:dataset> </dsi:datasets> <dsi:diagnostics> ... </dsi:diagnostics> </dsi:inventory> </syntaxhighlight> The dataset inventory will list all datasets published by a given data source, each with its title and ID. If archives exist for this dataset, they will be listed in ''archives'', each ''archive'' element storing the URL for the respective file. For XML archives, the attribute ''namespace'' will store the namespace of the schema, for DarwinCore archives there will an attribute ''rowType'' with the value <nowiki>http://rs.tdwg.org/dwc/terms/Occurrence</nowiki>. The attribute ''rcount'' specifies the number of records stored in the archive file, ''filesize'' and ''modified'' are hopefully self-explanatory. You can also view the dataset inventory using the manual QueryForm: Scroll down to the section ''Configuration Info'' and click on the link ''Inventory''. After a second, it will appear in the text box below. 1100e8bd1b50409d66d94f1146973eaebb50fe8b 842 841 2015-08-04T13:31:11Z JoergHoletschek 3 wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later.<br>If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousand or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The OpenUp! natural history aggregator, for example, can use this method for fast harvesting of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==Managing Archives== The section <small>Existing Archives</small> allows you download, remove and convert archives or view the log of the archiving process: [[File:xmlArchivesNew.png|border|775px]] The archives are grouped by schema, with DarwinCore archives making up an additional group. You can use the minus or plus symbol next to the group header to expand or collapse all files of that group; for web services publishing dozens of datasets, this might be handy to reduce scrolling on the page. The <small>Log </small> link in the group header will open the log file for the respective archiving process in a separate tab. For each archive file, the number of records stored in this file, the date of creation and the file size are listed. XML archives may have a number printed in the <small>Dropped</small> column, meaning some records have been dropped due to missing mandatory data elements. This doesn't mean the archive is invalid, but you should review the mapping or your source data to find the reasons for these lost records. Clicking the archive name will download the archive file, so it can be saved to disk or opened with an archive viewing tool. Pressing the trash symbol at the very right will delete the archive; if you use the trash symbol in the group header, all archives of this schema/type will be removed. For ABCD 2.06 and HISPID 5 XML archives, you will see an additional <small>DwC</small> link next to the trash symbol. This allows you to selectively convert a single dataset from XML to DarwinCore format. In contrast to using the <small>Create DwC Archive</small> button described below, this will spare the XML archiving of the whole web service and just create a DarwinCore archive for the selected dataset, which will be significantly faster. If you use the link in the group header, all archives of the schema will be converted (again without triggering XML archiving first). ==XML Archives== If you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. The section <small>Create/Update Archives</small> allows you to create new XML archive(s) for each schema mapped. Simply select the schema and press <small>Create XML Archive</small>. As mentioned above, archiving ABCD 2.06 or HISPID 5 is dataset-aware, resulting in one XML archive per dataset published. For all other schemas, a single archive file will be created. The paging size specifies how many records will be bundled into one XML document inside the archive; the default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgressNew.png|border]] You can always cancel the archiving process by pressing the <small>Cancel</small> link in the Status/Messages box. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Existing Archives</small> list, or, if it’s still being processed, as <small>Processing</small> in the Status/Messages box. If the process failed for some reason (or you decided to cancel it), a new box <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create XML archives for one schema at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. DarwinCore archives store only one dataset; so as with ABCD 2.06 and HISPID 5 XML archives, DwC-archiving a web service might result in several DarwinCore archives. DarwinCore archives are not created directly, but from the respective ABCD/HISPID archives. So DwC-archiving a web service will first create ACBD archives, which will be converted into the respective DwC archives subsequently. ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the library test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive before, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the archive name listed under ''Existing Archives''. The XML archiving step and the parameter <small>Paging size</small> are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgressNew.png|border]] You can cancel the transformation process by clicking the <small>Cancel</small> link in the Status/Messages box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from the ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: Conversion finished.'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. This parameter corresponds to the paging size that can be set on the archiving page. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. == Discovery of Archives == Archives available for a given data source can be discovered through the so-called dataset inventory. To get this, just append ''&inventory=1'' to the access URL of the data source, for example http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=BoBO&inventory=1. The dataset inventory response looks roughly like this (diagnostic messages have been removed for brevity): <syntaxhighlight> <dsi:inventory xsi:schemaLocation="http://www.biocase.org/schemas/dsi/1.0 http://www.bgbm.org/biodivinf/schema/dsi_1_0.xsd"> <!--XML generated by BioCASE PyWrapper software version 3.5.3. Made in Berlin.--> <dsi:status>OK</dsi:status> <dsi:created>2015-06-25T15:22:51.805000</dsi:created> <dsi:service_url>http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=BoBO</dsi:service_url> <dsi:datasets> <dsi:dataset> <dsi:title>BoBO - Botanic Garden and Botanical Museum Berlin-Dahlem Observations</dsi:title> <dsi:id>BoBO - Botanic Garden and Botanical Museum Berlin-Dahlem Observations</dsi:id> <dsi:archives> <dsi:archive filesize="143137" modified="2015-05-06T16:06:19.346807" namespace="http://www.tdwg.org/schemas/abcd/2.06" rcount="823">http://ww3.bgbm.org/biocase/downloads/BoBO/BoBO%20-%20Botanic%20Garden%20and%20Botanical%20Museum%20Berlin-Dahlem%20Observations.ABCD_2.06.zip</dsi:archive> <dsi:archive filesize="128502" modified="2015-05-06T16:06:34.768532" rcount="823" rowType="http://rs.tdwg.org/dwc/terms/Occurrence">http://ww3.bgbm.org/biocase/downloads/BoBO/BoBO%20-%20Botanic%20Garden%20and%20Botanical%20Museum%20Berlin-Dahlem%20Observations.DwCA.zip</dsi:archive> </dsi:archives> </dsi:dataset> </dsi:datasets> <dsi:diagnostics> ... </dsi:diagnostics> </dsi:inventory> </syntaxhighlight> The dataset inventory will list all datasets published by a given data source, each with its title and ID. If archives exist for this dataset, they will be listed in ''archives'', each ''archive'' element storing the URL for the respective file. For XML archives, the attribute ''namespace'' will store the namespace of the schema, for DarwinCore archives there will an attribute ''rowType'' with the value <nowiki>http://rs.tdwg.org/dwc/terms/Occurrence</nowiki>. The attribute ''rcount'' specifies the number of records stored in the archive file, ''filesize'' and ''modified'' are hopefully self-explanatory. You can also view the dataset inventory using the manual QueryForm: Scroll down to the section ''Configuration Info'' and click on the link ''Inventory''. After a second, it will appear in the text box below. 9da3b58325071a0973a8a49adf222074b330f002 0 52 863 797 2015-11-10T08:59:21Z JoergHoletschek 3 /* I cannot meet all of the requirements for installing the BPS. What options do I have? */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/ipt) you can create a DarwinCore Archive (see http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/ for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I'm using Internet Information Server, and after creating the Virtual Directory for BioCASe, I get a “502.2 - Bad Gateway. The specified CGI application misbehaved by not returning a complete set of HTTP headers” error.=== This happens when you didn't correctly associate .cgi and .py files with Python. Make sure you've linked both files with Python and that the path contains the placeholders ''%s %s'' (see [[Installation#Internet_Information_Server|Linking IIS]]). ===I cannot find the installation package of the database connectivity module for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later of psycopg2. ===I'm trying to use BioCASe with Postgres; for this, I tried to install psycopg on Windows. But it doesn't work - either the install crashes, or it finishes, but the test libs page still reports psycopg2 as not installed. In the debug log I can find the error message "ImportError: DLL load failed (...)". What's wrong?=== We suspect this is caused by a bug in recent psycopg2 releases (find more [http://psycopg.lighthouseapp.com/projects/62710/tickets/20 here]). On our test installations, we managed to get round this issue by installing pycopg2 with administration rights. Do do this, right-click the installer file and choose "Run as administrator". ===What's the difference between the DBMS packages odbc_tsql and odbc_tsql_2012 and between pymssql/pymssql_2012, respectively?=== The dbmods ending in "2012" use the new OFFSET clause that was introduced in MS SQL Server 2012. This will make paging faster and increase performance for larger datasets. They work only with SQL Server 2012 or later, and if you use SQL Server 2012, you should use them for best performance. Dbmods without a trailing "2012" work for both older versions and SQL Server 2012, but will be slower in paging large datasets. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== See [[Installation#Updating_an_existing_BioCASe_Installation|Updating an existing BioCASe Installation]]. ===I've updated my installation from a version prior to 2.6. Now, when I enter the DB structure, some of the table and column names are gone, the same happens in the mapping editor.=== Starting with version 2.6, BioCASe retrieves table and column names from the database for some database management systems (namely MySQL, SQL Server, Postgres, Foxpro, Access and Oracle). In older versions, you had to enter table/column names manually, and when you did the setup with such an early version, you probably didn't care for upper/lower cases. Since some DB management systems are case-sensitive, BioCASe can't ignore the differences and won't find these technically incorrect names in the lists retrieved from the database. The respective drop down boxes will therefore be empty. If that happens, just select the correct names from the drop down lists. ===When I try to create an archive, I get a "Permission denied" error. Why is that?=== For creating archives, you need to grant write permissions for the folders ''archive'' and ''www/downloads'' of your BioCASe installation to Python. You can check if permissions are set correctly in the test libs page (Start --> Utilities --> Test Libs). ===I'm trying to install BioCASe on Ubuntu (or another Linux). Even though I've run the setup script and added the snippet to the Apache configuration file, I see source code when I try out my installation instead of the pages described in the documentation=== Apparently CGI is not enabled on your machine. Google how to enable CGI for your distribution and follow the instructions you'll find. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other only by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===I've mapped ABCD2. When testing the web service, I get a negative number for "RecordsDropped" in the response document. What does that mean?=== Same problem as above - it means you have an error in your mapping. Most probably you've mapped a non-repeatable ABCD concept to a table/column in your database that holds several entries per unit (occurrence) record. The BioCASe Provider Software will duplicate the respective unit records to be able to create valid ABCD documents - which is indicated by a negative number of "records dropped". Make sure the cardinality of your database relations match the structure (in respects of element repeatability) of ABCD to avoid this problem. ===I'm using pymssql to connect to a SQL Server database. Char(n) and Varchar(n) strings get truncated to 255 characters - what's wrong?=== This is known limitation of TDS protocol used by pymssql. Just change the data type for columns storing long string to TEXT or create a view that converts Char/Varchar columns to TEXT columns on the fly to work around this issue. ===I'm using pymssql to connect to a SQL Server database, BioCASe is running on a Linux distribution. When I do a Search or Scan request, I don't get any results (0 hits), even though there are definitely records in the database.=== That problem occurs on old versions of pymssql (1.0.2 or earlier) on Linux distributions. Update to version 2.0 or later to solve this. If you still get empty result sets, these posts might be helpful: * http://open-up.cybertaxonomy.africamuseum.be/forum_topic/important_if_you_use_biocase_bps_linux_platform_connect_ms_sql_server * http://code.google.com/p/pymssql/issues/detail?id=76. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===I am using BioCASe on a Windows Server. For some queries with the Local QueryForm, I get an error page that is obviously produced by the web server and not BioCASe. It reads "Error Code: 500 Internal Server Error. The request was rejected by the HTTP filter. Contact the server administrator. (12217)"=== This happens when BioCASe is run behind a Microsoft ISA Server/Microsoft Forefront Threat Management Gateway. To solve this problem, follow the instructions given in this article: http://support.microsoft.com/kb/837865. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. ===I'm using BioCASe in a Vagrant Virtual Box on MacOS with Synced Folders. When using the Manual Query Form or the Archiving page, I get an "ImportError: No module named wrapper.errorclasses."=== We don't know exactly what causes this, but apparently it happens when BioCASe is run in a Vagrant Virtual Box on MacOS and Vagrant's synced folder are used. We do have a workaround for this, but it is not easy. Please [[Special:Contact|contact us]] for assistance on this. ===I'm using BioCASe on a Windows server and odbc_tsql to connect to an MS SQL Server. Even though I do get results for a SCAN request, I cannot do a SEARCH (invalid document received).=== Please check your encoding settings in the DB connection configuration. If it is set to UTF_8, try Latin_1 instead. == Trouble Shooting == ===I’ve forgotten the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. fbbf71955640e968a05d9094c883261d81326243 864 863 2015-11-10T09:01:15Z JoergHoletschek 3 /* I cannot meet all of the requirements for installing the BPS. What options do I have? */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/ipt) you can create a DarwinCore Archive (see http://rs.tdwg.org/dwc/terms/guides/text/index.htm for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I'm using Internet Information Server, and after creating the Virtual Directory for BioCASe, I get a “502.2 - Bad Gateway. The specified CGI application misbehaved by not returning a complete set of HTTP headers” error.=== This happens when you didn't correctly associate .cgi and .py files with Python. Make sure you've linked both files with Python and that the path contains the placeholders ''%s %s'' (see [[Installation#Internet_Information_Server|Linking IIS]]). ===I cannot find the installation package of the database connectivity module for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later of psycopg2. ===I'm trying to use BioCASe with Postgres; for this, I tried to install psycopg on Windows. But it doesn't work - either the install crashes, or it finishes, but the test libs page still reports psycopg2 as not installed. In the debug log I can find the error message "ImportError: DLL load failed (...)". What's wrong?=== We suspect this is caused by a bug in recent psycopg2 releases (find more [http://psycopg.lighthouseapp.com/projects/62710/tickets/20 here]). On our test installations, we managed to get round this issue by installing pycopg2 with administration rights. Do do this, right-click the installer file and choose "Run as administrator". ===What's the difference between the DBMS packages odbc_tsql and odbc_tsql_2012 and between pymssql/pymssql_2012, respectively?=== The dbmods ending in "2012" use the new OFFSET clause that was introduced in MS SQL Server 2012. This will make paging faster and increase performance for larger datasets. They work only with SQL Server 2012 or later, and if you use SQL Server 2012, you should use them for best performance. Dbmods without a trailing "2012" work for both older versions and SQL Server 2012, but will be slower in paging large datasets. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== See [[Installation#Updating_an_existing_BioCASe_Installation|Updating an existing BioCASe Installation]]. ===I've updated my installation from a version prior to 2.6. Now, when I enter the DB structure, some of the table and column names are gone, the same happens in the mapping editor.=== Starting with version 2.6, BioCASe retrieves table and column names from the database for some database management systems (namely MySQL, SQL Server, Postgres, Foxpro, Access and Oracle). In older versions, you had to enter table/column names manually, and when you did the setup with such an early version, you probably didn't care for upper/lower cases. Since some DB management systems are case-sensitive, BioCASe can't ignore the differences and won't find these technically incorrect names in the lists retrieved from the database. The respective drop down boxes will therefore be empty. If that happens, just select the correct names from the drop down lists. ===When I try to create an archive, I get a "Permission denied" error. Why is that?=== For creating archives, you need to grant write permissions for the folders ''archive'' and ''www/downloads'' of your BioCASe installation to Python. You can check if permissions are set correctly in the test libs page (Start --> Utilities --> Test Libs). ===I'm trying to install BioCASe on Ubuntu (or another Linux). Even though I've run the setup script and added the snippet to the Apache configuration file, I see source code when I try out my installation instead of the pages described in the documentation=== Apparently CGI is not enabled on your machine. Google how to enable CGI for your distribution and follow the instructions you'll find. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other only by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===I've mapped ABCD2. When testing the web service, I get a negative number for "RecordsDropped" in the response document. What does that mean?=== Same problem as above - it means you have an error in your mapping. Most probably you've mapped a non-repeatable ABCD concept to a table/column in your database that holds several entries per unit (occurrence) record. The BioCASe Provider Software will duplicate the respective unit records to be able to create valid ABCD documents - which is indicated by a negative number of "records dropped". Make sure the cardinality of your database relations match the structure (in respects of element repeatability) of ABCD to avoid this problem. ===I'm using pymssql to connect to a SQL Server database. Char(n) and Varchar(n) strings get truncated to 255 characters - what's wrong?=== This is known limitation of TDS protocol used by pymssql. Just change the data type for columns storing long string to TEXT or create a view that converts Char/Varchar columns to TEXT columns on the fly to work around this issue. ===I'm using pymssql to connect to a SQL Server database, BioCASe is running on a Linux distribution. When I do a Search or Scan request, I don't get any results (0 hits), even though there are definitely records in the database.=== That problem occurs on old versions of pymssql (1.0.2 or earlier) on Linux distributions. Update to version 2.0 or later to solve this. If you still get empty result sets, these posts might be helpful: * http://open-up.cybertaxonomy.africamuseum.be/forum_topic/important_if_you_use_biocase_bps_linux_platform_connect_ms_sql_server * http://code.google.com/p/pymssql/issues/detail?id=76. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===I am using BioCASe on a Windows Server. For some queries with the Local QueryForm, I get an error page that is obviously produced by the web server and not BioCASe. It reads "Error Code: 500 Internal Server Error. The request was rejected by the HTTP filter. Contact the server administrator. (12217)"=== This happens when BioCASe is run behind a Microsoft ISA Server/Microsoft Forefront Threat Management Gateway. To solve this problem, follow the instructions given in this article: http://support.microsoft.com/kb/837865. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. ===I'm using BioCASe in a Vagrant Virtual Box on MacOS with Synced Folders. When using the Manual Query Form or the Archiving page, I get an "ImportError: No module named wrapper.errorclasses."=== We don't know exactly what causes this, but apparently it happens when BioCASe is run in a Vagrant Virtual Box on MacOS and Vagrant's synced folder are used. We do have a workaround for this, but it is not easy. Please [[Special:Contact|contact us]] for assistance on this. ===I'm using BioCASe on a Windows server and odbc_tsql to connect to an MS SQL Server. Even though I do get results for a SCAN request, I cannot do a SEARCH (invalid document received).=== Please check your encoding settings in the DB connection configuration. If it is set to UTF_8, try Latin_1 instead. == Trouble Shooting == ===I’ve forgotten the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. 01142234bdbffc31f393d0bcc18c889ea602e794 0 48 865 333 2015-11-10T09:06:16Z JoergHoletschek 3 wikitext text/x-wiki ;ABCD Data Schema (Access to Biological Collection Data): XML data schema used for publishing specimen collections and observational databases to primary biodiversity networks such as BioCASe and GBIF. The current version of ABCD is 2.06. ;Apache HTTP Server (“Apache”): Widely used open-source web server software. Apache is available for most operating systems and recommended for running BioCASe. ;BioCASe Installation: An installation of the BioCASe Provider Software that offers one or several BioCASe Web Services. ;BioCASe Web Service: A web service offered by a BioCASe installation. A BioCASe web service provides access to the database to be published and can be queried using the BioCASe Protocol. It is uniquely identified by its URL, which is a combination of the BioCASe installation’s URL and the name of the data source, for example http://www.myinstitution.org/biocase/pywrapper.cgi?dsa=Herbar. See [[DatasourceSetup]] for more information on how to set up a BioCASe Web Service. ;BioCASe (Biological Collection Access Service) Network: A network of primary biodiversity data providers. Currently, BioCASe offers three different data portals: One on the German flora (http://search.biocase.de/botany), one for the biodiversity of Europe (http://search.biocase.org/europe) and one for global occurrence data (http://search.biocase.org/edit). ;BioCASE Project (Biological Collection Access Service for Europe): EU-funded project (2002-05) that developed standards and software for publishing primary biodiversity data (specimen collections and observational databases). Within the first years, the results comprised the BioCASe Protocol, the ABCD data schema and the BioCASe Provider Software. ;BioCASe Protocol: The protocol for querying a BioCASe web service. The protocol defines three types of requests: ''Capabilities'' for introspecting the configuration of a BioCASe web service, ''Scan'' for getting the distinctive values for a concept provided by the web service, and ''Search'' for retrieving the full set of information published for certain records. ;BioCASe Request: See BioCASe Protocol. ;BioCASe Technology (Biological Collection Access Service): The suite of standards and software developed by the BioCASE Project and its successors. Currently, the main components are the BioCASe Protocol, the ABCD data schema, BioCASe Provider Software, the SYNTHESYS Cache Generation System and the BioCASe Data Portal Software. ;BPS: BioCASe Provider Software; also called PyWrapper. ;Cache: A component (in the BioCASe context, usually a database or several database tables) that temporarily stores data from another source. In most cases, this is done in order to increase performance for information retrieval. ;Configuration tool (also Config Tool): Set of dialogs used for configuring a BioCASe installation or BioCASe web services. ;Database View: A virtual table in a database that offers another view on the data stored in the database. A view can combine data from several tables and transform, that is, translate, concatenate or pivot them. Updates of the underlying tables are always reflected in views drawing data from these tables. ;Datasource: In the BioCASe context, a datasource is database that is to be published as a BioCASe web service. Do not confuse that with an ODBC datasource that defines the connection parameters for connecting to a database through ODBC. ;DBMS: Database Management System. ;Debugging: The iterative process of testing, finding and correcting errors. ;Denormalisation: In a relational database, the information for real world objects are split into several tables, which is called normalisation. Denormalisation is the process of intentionally reversing this step for the sake of faster information retrieval. ;Firewall: Usually one device, a combination of several devices or a software shielding a local area network (LAN) from the internet. A firewall filters all network traffic between the LAN and the Internet, blocking unwanted protocols (i.e. applications) and ports. In a BioCASe context, servers should be placed in a way that no firewall is located between the web server running the Provider Software and the database server. In case there is a firewall in between, the port used by the database management system (e.g. 3306 for MySQL) must be opened. ;Global Biodiversity Information Facility (GBIF): International organisation for promoting free and universal access to worldwide primary biodiversity information. One of the main services offered is the GBIF Data Portal (http://www.gbif.org). ;Harvester: Application that retrieves all records published by a (BioCASe) web service and stores them in a local database. Harvesters are run by projects and initiatives that consume or aggregate primary biodiversity information, for example by GBIF for running the [http://www.gbif.org GBIF data portal]. ;Identifier: In a database table, an identifier is a field that names a record in that table. Identifiers are usually unique, that means the identifier is different for each record. ;Internet Information Services (IIS): Web service platform integrated into Microsoft’s server operating systems (Server 2000, 2003 and 2008); can be used for running the BioCASe Provider Software. ;Local Query Tool: Component of the BioCASe Provider Software that can be used to access a BioCASe web service. The Query Tool offers a customizable, very simple and no-frills search interface for querying '''one''' (!) BioCASe web service. ;Mapping Editor: Page of the configuration tool for mapping concepts of the published data schema to the source tables/columns in the database. ;Metadata: Data about data. In the BioCASe context, this is a set of information describing the database published as a whole - title, description, curator’s name, ownership, date of last modification, terms of use etc. ;Open Database Connectivity (ODBC): Standard Interface for accessing relational database management systems. The connection parameters for connecting to a database through ODBC are defined in an ODBC Datasource. ;OS: Operating System (Windows, MacOS, Ubuntu, Debian, Ret Hat, Solaris etc.). ;Python: Platform independent programming language used for the BioCASe Provider Software. ;PyWrapper: Synonym for the BioCASe Provider Software (BPS), created from "Python Wrapper". Sort of "wraps" around the database to be published, exposing the data as a BioCASe compliant web service. ;Query Form: Component of the Provider Software for sending requests to a BioCASe web service and displaying the result documents; usually used for debugging a web service. ;Schema Mapping: The part of the configuration that tells the Provider Software how to create documents in a specific published data schema from the source database. A BioCASe web service can support several schemas (e.g. for ABCD and DarwinCore) by having several schema mappings. ;SQL (Structured Query Language): Language for managing data in relational database management systems, e.g. for information retrieval. ;Subversion: Software versioning and revision control system used for the BioCASe Provider Software. When installing BioCASe, the software can be downloaded directly from the Subversion repository (See [[Installation#Downloading from the Subversion Repository]]). ;Wrapper: In the BioCASe context a synonym for PyWrapper. c776f9a7513511e1cbca9ef1e579670f76456b2e 2 73 866 290 2015-11-10T14:35:48Z DavidFichtmueller 4 testing dead link finder javascript text/javascript var deadLinkFinder_showOk = true; var deadLinkFinder_fadeOk = true; var deadLinkFinder_showBrowsemodeLink = true; var deadLinkFinder_showWaitingIcon = true; //var deadLinkFinder_language = "en"; var deadLinkFinder_namespaceFilter = [0,2,4,6,8,12,14,100]; //var deadLinkFinder_runAlways = true; var deadLinkFinder_linkToLinkSearch = true; var deadLinkFinder_showIndividualOK = true; var deadLinkFinder_showRedirects = true; var deadLinkFinder_showContentType = true; mw.loader.load("//biowikifarm.org/tools/deadlinkfinder/script.js"); addOnloadHook(function(){ addPortletLink("p-personal","http://wiki.bgbm.org/bps/index.php?title=User:DavidFichtmueller/monobook.js&action=edit","monobook.js", "t-edit-vector","edit vector.js","v",document.getElementById("pt-preferences")); var firstElement = document.getElementById("t-specialpages").parentNode.firstElementChild; addPortletLink("p-tb","http://wiki.bgbm.org/bps/index.php/Special:RecentChanges","Recent Changes", "t-recent-changes","Recent Changes","s",firstElement); if(wgNamespaceNumber==2 || wgNamespaceNumber==3){ //user namespace or user discussion namespace var slashPos = wgTitle.indexOf('/'); var username = wgTitle; if(slashPos != -1){ // it's a subpage username = username.substring(0,slashPos); } addPortletLink("p-cactions","http://wiki.bgbm.org/bps/index.php/Special:Contributions/"+username,"Contributions", "ca-contributions","Contributions of "+username,"b",document.getElementById("ca-talk").nextElementSibling); document.getElementById("ca-talk").style.marginRight="0.3em"; document.getElementById("ca-contributions").style.marginRight="1.6em"; } } ); //closes addOnloadHook( 6584d50a3101cedccd3aeb4e0ca57cb36b58c119 2 111 867 2015-11-12T10:10:54Z JoergHoletschek 3 Created page with "var deadLinkFinder_showOk = true; var deadLinkFinder_fadeOk = true; var deadLinkFinder_showBrowsemodeLink = true; var deadLinkFinder_showWaitingIcon = true; //var deadLinkFind..." javascript text/javascript var deadLinkFinder_showOk = true; var deadLinkFinder_fadeOk = true; var deadLinkFinder_showBrowsemodeLink = true; var deadLinkFinder_showWaitingIcon = true; //var deadLinkFinder_language = "en"; var deadLinkFinder_namespaceFilter = [0,2,4,6,8,12,14,100]; //var deadLinkFinder_runAlways = true; var deadLinkFinder_linkToLinkSearch = true; var deadLinkFinder_showIndividualOK = true; var deadLinkFinder_showRedirects = true; var deadLinkFinder_showContentType = true; mw.loader.load("//biowikifarm.org/tools/deadlinkfinder/script.js"); b7add6220fdda751f5b742e4b035b81ea15ca70f 2 112 868 2015-11-12T10:34:32Z JoergHoletschek 3 Created page with "#p-tb{ display:block !important; }" css text/css #p-tb{ display:block !important; } 651a2024c9dbcdfbe422ddd06f1722b356804b98 0 49 871 847 2015-11-26T12:33:22Z JoergHoletschek 3 /* Support */ wikitext text/x-wiki [[http://wiki.bgbm.org/bps/index.php/BeginnersGuide_de Diese Seite auf Deutsch]] == What is the BioCASe Provider Software? == The BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. The Provider Software is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. The Provider Software is a suite of tools that need to be installed on the data provider’s web server. The core component is the PyWrapper, an XML/CGI database interface written in Python that allows a standardized access to a variety of database management systems and arbitrarily structured databases. Grouped around this are a number of tools for configuring, testing and debugging a BioCASe installation or web service. Even though BioCASe can be used for any conceptual XML data schema, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the [http://www.biocase.org BioCASe network] and the [http://www.gbif.org Global Biodiversity Information Facility]. == Requirements for Using the BPS == Before using the BioCASe Provider Software, four requirements need to be checked: * For one thing, the data to be published needs to be stored in an SQL compliant relational database management system (DBMS), with the data model being well documented or at least knowledge about the data model existing on someone’s head. Non-SQL DBMS are not supported. Currently, the BioCASe Provider Software can connect to the following databases: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * For installing BioCASe, you need to have a web server (Apache or Microsoft IIS) that is permanently connected to the Internet. This can be an institutional server or a hosting service (even though it’s usually a pain to get privileges to install software for the latter). Dial-up connections won’t work. * For the web server in question, you must have the possibility to install Python (if it is not installed yet) and Python packages. Even though this can be done by the IT staff, you should make sure that there are no institutional policies colliding. If you want to create DarwinCore Archives, a Java Runtime Environment is required. * A direct connection can be established from the web server that will run the BPS to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL, 1433 for SQL Server). == Steps == Once you’ve made sure the BioCASe Provider Software is the right choice for you ([[Special:Contact|contact the BioCASe team]] if you feel unsafe with your decision), you need to do the following steps: # Think about which information you want to publish to the network. This involves settling the question of ownership of the data and the terms of use, license and usage restrictions for the published data. Also you should consider which information should '''not''' be published because they might affect endangered species (options are to block certain occurrences from publication or to blur the exact locality information). # Prepare the database for publication as described in the [[Preparation]] tutorial. This includes the decision on whether to publish the live database or a snapshot, the creation of a table for metadata and, in certain cases, some data transformation. # Follow the [[Installation]] Tutorial to install the BioCASe Provider Software. # Set up a BioCASe Datasource and connect it to your database, then configure the database structure as described in the [[DatasourceSetup]] tutorial. # Follow the [[ABCD2Mapping]] tutorial to create a mapping for the schema you want to use. # Test and debug your BioCASe web service. Read the [[Debugging]] tutorial to learn how to do this. # Once you’re done with setting up and configuring the BioCASe Provider Software, you can register the BioCASe web service with the information network you want to publish your data to, for example with [http://www.gbif.org GBIF]. We strongly recommend to [[Special:Contact|contact the BioCASe team]] for checking your web service first as we might be able to foresee problems that are more difficult to resolve after registration. == Support == In case you run into difficulties during the installation and setup, you should * have a look at the [[FAQ]] which might list the solution for your question, * ask other BioCASe data providers for help, * get in contact with the [[Special:Contact|BioCASe team]]. Please describe exactly your environment (operating system, DBMS used), the steps that cause the problem and any error messages and hints you get. Screenshot are a big plus here. f869d7a8e776050c8dbc89101c70e5b2d6afced6 0 23 872 870 2016-01-18T13:40:24Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * New dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 3f09d0cac3b769d6c7c4adcc58fa78ede00c7211 876 872 2016-03-14T16:16:50Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * New dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 4296f6bf1b57ac336e40fa9ace509102755e32a8 877 876 2016-03-14T16:22:24Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool c4eff6d90061228d1bd216f226f9c25d851a61da 878 877 2016-03-14T16:38:35Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 8aae63af500ef12d74be8bca82e975cd09bc2f80 879 878 2016-03-31T14:40:28Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Bugfix for DarwinCore transformation (one identification lost when identical scientific names were provided as both a preferred and a non-preferred identification) === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 2b6e35a18eeda5a1b50aeac9eacac23799ed2a25 880 879 2016-04-01T12:57:25Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Bugfixes for DarwinCore transformation: ** One of the identifications lost if identical scientific names were provided for both a preferred and a non-preferred identification ** HigherClassification contained double entries if higher taxa were provided for identifications with empty scientific name === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 6ea921406801a57872c6f1531276882512203321 881 880 2016-04-07T11:22:53Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Changes in DarwinCore transformation: ** Transformation streamlined ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 9ab5432e7e04a3a479b72acf5d806301eff64eae 882 881 2016-04-12T15:45:30Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Changes in DarwinCore transformation: ** Transformation streamlined ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Bugfix in metadata retrieval for dbmod_pymssql. === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 7c0416aceaab4b293c4cca84e056effb96dea127 883 882 2016-05-19T12:24:17Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Bugfix in metadata retrieval for dbmod_pymssql. === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool b6e6b30b6f6d047391bed98358ac82dece9c0248 884 883 2016-06-16T07:38:02Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Bugfix in metadata retrieval for dbmod_pymssql. === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool db407cdc597ed2b43e634ba8b528d2380141547b 885 884 2016-06-16T09:50:21Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool bd3cb26e874488119ccb6d7d48facaf64af8ff2b 886 885 2016-07-21T09:05:16Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Version 3.6.1 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool a3901efded4366187a33a5c18d9ee19b6ae00f3e 887 886 2016-07-21T09:05:25Z JoergHoletschek 3 /* Version 3.6.1 [2016-07-21] */ wikitext text/x-wiki == Current Version 3 == === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool afc2b0482cbdc51f5799d463abb8bb265f872f05 888 887 2016-07-21T13:21:05Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Small bugfix for dbmod_oracle === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 2c9d78ebd6617214c32588fe1139b9d7a4f0ed04 892 888 2016-07-28T14:38:03Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Fix/change for dbmod_oracle.py: Paging is now faster === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool d748edd42706e2cfb62ab77190b1754ecfcba89d 894 892 2016-08-04T10:52:28Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * DwC arhive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool b56f4648d041f0e5284f2bb64a64ce4788f32fc9 896 894 2016-08-08T08:52:36Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool ab01d3d95bb14302afa01835b23d9eac4d39c8f2 897 896 2016-08-08T16:22:02Z JoergHoletschek 3 /* Upcoming [2016-08-08] */ wikitext text/x-wiki == Current Version 3 == === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 8bc12f962293bdce1a00d0ec25fa888f09842661 901 897 2016-10-21T14:29:49Z JoergHoletschek 3 /* Version 3.6.3 [2016-08-08] */ wikitext text/x-wiki == Current Version 3 == === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool a6fc00563152987e6d68e373fa7a2a699317104b 902 901 2016-11-09T11:50:02Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Bugfix for "remove cached objects" in system administration === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 3cd0bb2b2b040cae607a1189a0b333da1c2c827b 903 902 2016-12-12T10:27:21Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Fixes ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 4359bbd45885bd1c984cfd762359ab26bebd730b 904 903 2017-03-12T17:20:19Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool afc91361e0bcb434e74a7e577d9d08745d78219a 905 904 2017-04-04T12:46:00Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool f2c26c87f4be37a576ea70e4baab92af0630c468 906 905 2017-05-03T12:14:01Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 927b323d33e014541d9ecb5bca1c82d241d4f6b8 907 906 2017-05-16T15:58:32Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool effa72a037f46df5c0f212cc6fed4943cfe063c0 908 907 2017-05-22T15:27:02Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * QueryTool now lists annotations created with [http://www.example.com AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 60259631493a6ad63b5d7203fac7e1b65920a89f 909 908 2017-07-04T12:51:24Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [http://www.example.com AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool e098b118a12705e577c667d8c8499f9502499bde 910 909 2017-08-17T13:42:55Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Bugfix in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [http://www.example.com AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 5ad51352162edecb0b28f0121f4cbbd42ff38ad4 911 910 2017-08-28T10:25:01Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Fixes ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [http://www.example.com AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool f9324a13c45d74dae0fd61d2b7d506dbb843b0c6 912 911 2017-09-13T12:49:41Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Fixes ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [http://www.example.com AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 0a4a88b8dad5f4ff6b5235695dcff5f44a8281b5 913 912 2017-09-26T10:08:13Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example openjre) * Fixes ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [http://www.example.com AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 959ea918f8256916a9982e33fbdc388d5f486e1b 914 913 2017-09-26T10:27:57Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * Fixes ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [http://www.example.com AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool b48181a2f74cc0d88379ee333a5e28ae21bb3413 918 914 2017-11-16T10:40:19Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * Fixes ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [http://www.example.com AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool a5317faa9f5d5066abaaef734ccc9803417b847d 919 918 2017-11-22T14:37:14Z JoergHoletschek 3 /* Version 3.6.4 [2017-07-04] */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * Fixes ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 4bad3ac3e8f96dad31002c38b5de390954b2c29c 920 919 2017-12-06T14:49:20Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * Fixes ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool e398e86093b7ff4d02cfb95b87ab25c95dea5548 2 113 873 2016-03-05T21:58:18Z 178.150.15.176 0 Good say's! wikitext text/x-wiki Perfectly written! I will right away grab your rss as I can not to find your email subscription hyperlink or e-newsletter service. Do you’ve any? Kindly permit me realize in order that I may subscribe. 9828e55c073439d156545ae863420b1a53d9a158 874 873 2016-03-14T10:54:17Z JoergHoletschek 3 wikitext text/x-wiki Perfectly written! I will right away grab your rss as I can not to find your email subscription hyperlink or e-newsletter service. Do you’ve any? Kindly permit me realize in order that I may subscribe. --------------- Dear user, we don't have news letter. But all news regarding the BioCASe Provider Software will be announced on the RSS feed available here: http://www.biocase.org/rss.xml If you have any further questions, please use the contact form or the email address given on that page. Cheers, Jörg 6214a935312ec2ed709ad79862b6e7675a6b0dbd 875 874 2016-03-14T10:54:51Z JoergHoletschek 3 wikitext text/x-wiki Perfectly written! I will right away grab your rss as I can not to find your email subscription hyperlink or e-newsletter service. Do you’ve any? Kindly permit me realize in order that I may subscribe. --------------- Dear user, we don't have news letter. But all news regarding the BioCASe Provider Software will be announced on the RSS feed available here: http://www.biocase.org/rss.xml If you have any further questions, please use the contact form or the email address given on that page: http://wiki.bgbm.org/bps/index.php/Special:Contact Cheers, Jörg afe006b2310ceacfc81a8cb88f3f67679b37800b 0 22 889 700 2016-07-27T09:12:12Z JoergHoletschek 3 /* Highly Recommended */ wikitext text/x-wiki This page lists the most commonly used ABCD 2.06 concepts. The ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a full list of ABCD concepts [http://wiki.tdwg.org/twiki/bin/view/ABCD/AbcdConcepts here]. == Mandatory == List of mandatory concepts. Without them being mapped the Provider Software will not be able to create valid ABCD document: <pre> DataSets └─ DataSet ├─ ContentContacts │ └─ ContentContact │ └─ Name ├─ TechnicalContacts │ └─ TechnicalContact │ └─ Name ├─ Metadata │ ├─ Description │ │ └─ Representation │ │ ├─ @language │ │ └─ Title │ └─ RevisionData │ └─ DateModified └─ Units └─ Unit ├─ SourceInstitutionID ├─ SourceID └─ UnitID </pre> == Highly Recommended == List of recommended ABCD concepts. If you can provide this data, please do so, they're commonly used entry points to your data. <pre> DataSets └─ DataSet ├─ ContentContacts │ └─ ContentContact │ ├─ Email │ └─ Address ├─ TechnicalContacts │ └─ TechnicalContact │ ├─ Email │ └─ Address ├─ Metadata │ ├─ Description │ │ └─ Representation │ │ └─ Details │ └─ Owners │ └─ Owner │ ├─ LogoURI │ ├─ URIs │ │ └─ URL │ ├─ Person │ │ └─ FullName │ └─ Organisation │ └─ Name │ └─ Representation │ └─ Text ├─ IPRStatements │ └─ Citations │ └─ Citation │ ├─ @language │ ├─ Text │ └─ URI └─ Units └─ Unit ├─ Identifications │ └─ Identification │ └─ Result │ ├─ @PreferredFlag │ ├─ TaxonIdentified │ │ └─ ScientificName │ │ └─ FullScientificNameString │ └─ HigherTaxa │ └─ HigherTaxon │ ├─ HigherTaxonName │ └─ HigherTaxonRank ├─ RecordBasis └─ Gathering ├─ DateTime │ └─ ISODateTimeBegin ├─ Agents │ └─ GatheringAgent │ └─ Person │ └─ FullName ├─ LocalityText ├─ Country │ ├─ ISO3166Code │ └─ Name └─ SiteCoordinateSets └─ SiteCoordinates └─ CoordinatesLatLong ├─ LongitudeDecimal ├─ LatitudeDecimal └─ CoordinateErrorDistanceInMeters </pre> == Commonly Used == List of further commonly mapped ABCD concepts. <pre> DataSets └─ DataSet ├─ Metadata │ └─ IPRStatements │ ├─ TermsOfUseStatements │ │ └─ TermsOfUse │ │ └─ Text │ ├─ Copyrights │ │ └─ Copyright │ │ └─ Text │ ├─ IPRDeclarations │ │ └─ IPRDeclaration │ │ └─ Text │ ├─ Licenses │ │ └─ License │ │ └─ Text │ ├─ Disclaimers │ │ └─ Disclaimer │ │ └─ Text │ ├─ Citations │ │ └─ Citation │ │ └─ Text │ └─ Acknowledgements │ └─ Acknowledgement │ └─ Text └─ Units └─ Unit ├─ DateLastEdited ├─ Identifications │ └─ Identification │ ├─ Date │ │ └─ ISODateTimeBegin │ └─ Identifiers │ └─ Identifier │ └─ IdentifierPersonName │ └─ FullName ├─ KindOfUnit ├─ CollectorsFieldNumber ├─ Sex ├─ Age ├─ Notes ├─ RecordURI ├─ MultiMediaObjects │ └─ MultiMediaObject │ ├─ FileURI │ ├─ Comment │ └─ Context ├─ SpecimenUnit │ └─ NomenclaturalTypeDesignations │ └─ NomenclaturalTypeDesignation │ ├─ TypeStatus │ ├─ TypifiedName │ │ └─ FullScientificNameString │ └─ NomenclaturalReference │ ├─ TitleCitation │ └─ CitationDetail └─ Gathering ├─ Project │ └─ ProjectTitle ├─ Biotope │ └─ Name ├─ Altitude │ └─ MeasurementOrFactAtomised │ ├─ LowerValue │ └─ UnitOfMeasurement ├─ Depth │ └─ MeasurementOrFactAtomised │ ├─ LowerValue │ └─ UnitOfMeasurement └─ NamedAreas └─ NamedArea ├─ AreaName └─ AreaClass </pre> 6c5808a0945259f9e454015074dc60764b0161b0 890 889 2016-07-27T09:18:23Z JoergHoletschek 3 /* Highly Recommended */ wikitext text/x-wiki This page lists the most commonly used ABCD 2.06 concepts. The ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a full list of ABCD concepts [http://wiki.tdwg.org/twiki/bin/view/ABCD/AbcdConcepts here]. == Mandatory == List of mandatory concepts. Without them being mapped the Provider Software will not be able to create valid ABCD document: <pre> DataSets └─ DataSet ├─ ContentContacts │ └─ ContentContact │ └─ Name ├─ TechnicalContacts │ └─ TechnicalContact │ └─ Name ├─ Metadata │ ├─ Description │ │ └─ Representation │ │ ├─ @language │ │ └─ Title │ └─ RevisionData │ └─ DateModified └─ Units └─ Unit ├─ SourceInstitutionID ├─ SourceID └─ UnitID </pre> == Highly Recommended == List of recommended ABCD concepts. If you can provide this data, please do so, they're commonly used entry points to your data. If you publish your data to GBIF, you are required to specify a license in the ''Citation'' node with the ''language'' attribute and the elements ''Text'' and ''URI''. The latter should contain a link to a well-defined license, for example https://creativecommons.org/publicdomain/zero/1.0/. More details on GBIF licensing can be found [http://www.gbif.org/terms/licences here]. <pre> DataSets └─ DataSet ├─ ContentContacts │ └─ ContentContact │ ├─ Email │ └─ Address ├─ TechnicalContacts │ └─ TechnicalContact │ ├─ Email │ └─ Address ├─ Metadata │ ├─ Description │ │ └─ Representation │ │ └─ Details │ └─ Owners │ └─ Owner │ ├─ LogoURI │ ├─ URIs │ │ └─ URL │ ├─ Person │ │ └─ FullName │ └─ Organisation │ └─ Name │ └─ Representation │ └─ Text ├─ IPRStatements │ └─ Citations │ └─ Citation │ ├─ @language │ ├─ Text │ └─ URI └─ Units └─ Unit ├─ Identifications │ └─ Identification │ └─ Result │ ├─ @PreferredFlag │ ├─ TaxonIdentified │ │ └─ ScientificName │ │ └─ FullScientificNameString │ └─ HigherTaxa │ └─ HigherTaxon │ ├─ HigherTaxonName │ └─ HigherTaxonRank ├─ RecordBasis └─ Gathering ├─ DateTime │ └─ ISODateTimeBegin ├─ Agents │ └─ GatheringAgent │ └─ Person │ └─ FullName ├─ LocalityText ├─ Country │ ├─ ISO3166Code │ └─ Name └─ SiteCoordinateSets └─ SiteCoordinates └─ CoordinatesLatLong ├─ LongitudeDecimal ├─ LatitudeDecimal └─ CoordinateErrorDistanceInMeters </pre> == Commonly Used == List of further commonly mapped ABCD concepts. <pre> DataSets └─ DataSet ├─ Metadata │ └─ IPRStatements │ ├─ TermsOfUseStatements │ │ └─ TermsOfUse │ │ └─ Text │ ├─ Copyrights │ │ └─ Copyright │ │ └─ Text │ ├─ IPRDeclarations │ │ └─ IPRDeclaration │ │ └─ Text │ ├─ Licenses │ │ └─ License │ │ └─ Text │ ├─ Disclaimers │ │ └─ Disclaimer │ │ └─ Text │ ├─ Citations │ │ └─ Citation │ │ └─ Text │ └─ Acknowledgements │ └─ Acknowledgement │ └─ Text └─ Units └─ Unit ├─ DateLastEdited ├─ Identifications │ └─ Identification │ ├─ Date │ │ └─ ISODateTimeBegin │ └─ Identifiers │ └─ Identifier │ └─ IdentifierPersonName │ └─ FullName ├─ KindOfUnit ├─ CollectorsFieldNumber ├─ Sex ├─ Age ├─ Notes ├─ RecordURI ├─ MultiMediaObjects │ └─ MultiMediaObject │ ├─ FileURI │ ├─ Comment │ └─ Context ├─ SpecimenUnit │ └─ NomenclaturalTypeDesignations │ └─ NomenclaturalTypeDesignation │ ├─ TypeStatus │ ├─ TypifiedName │ │ └─ FullScientificNameString │ └─ NomenclaturalReference │ ├─ TitleCitation │ └─ CitationDetail └─ Gathering ├─ Project │ └─ ProjectTitle ├─ Biotope │ └─ Name ├─ Altitude │ └─ MeasurementOrFactAtomised │ ├─ LowerValue │ └─ UnitOfMeasurement ├─ Depth │ └─ MeasurementOrFactAtomised │ ├─ LowerValue │ └─ UnitOfMeasurement └─ NamedAreas └─ NamedArea ├─ AreaName └─ AreaClass </pre> 576409b62c5fa13a2fbdb4ea73536d6eab5928ea 891 890 2016-07-27T09:27:00Z JoergHoletschek 3 /* Highly Recommended */ wikitext text/x-wiki This page lists the most commonly used ABCD 2.06 concepts. The ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a full list of ABCD concepts [http://wiki.tdwg.org/twiki/bin/view/ABCD/AbcdConcepts here]. == Mandatory == List of mandatory concepts. Without them being mapped the Provider Software will not be able to create valid ABCD document: <pre> DataSets └─ DataSet ├─ ContentContacts │ └─ ContentContact │ └─ Name ├─ TechnicalContacts │ └─ TechnicalContact │ └─ Name ├─ Metadata │ ├─ Description │ │ └─ Representation │ │ ├─ @language │ │ └─ Title │ └─ RevisionData │ └─ DateModified └─ Units └─ Unit ├─ SourceInstitutionID ├─ SourceID └─ UnitID </pre> == Highly Recommended == List of recommended ABCD concepts. If you can provide this data, please do so, they're commonly used entry points to your data. '''Note:''' If you publish your data to GBIF, you are required to specify a license in the ''Citation'' node with the ''language'' attribute and the elements ''Text'' and ''URI''. The latter should contain a link to a well-defined license, for example https://creativecommons.org/publicdomain/zero/1.0/. More details on GBIF licensing can be found [http://www.gbif.org/terms/licences here]. <pre> DataSets └─ DataSet ├─ ContentContacts │ └─ ContentContact │ ├─ Email │ └─ Address ├─ TechnicalContacts │ └─ TechnicalContact │ ├─ Email │ └─ Address ├─ Metadata │ ├─ Description │ │ └─ Representation │ │ └─ Details │ └─ Owners │ └─ Owner │ ├─ LogoURI │ ├─ URIs │ │ └─ URL │ ├─ Person │ │ └─ FullName │ └─ Organisation │ └─ Name │ └─ Representation │ └─ Text ├─ IPRStatements │ └─ Citations │ └─ Citation │ ├─ @language │ ├─ Text │ └─ URI └─ Units └─ Unit ├─ Identifications │ └─ Identification │ └─ Result │ ├─ @PreferredFlag │ ├─ TaxonIdentified │ │ └─ ScientificName │ │ └─ FullScientificNameString │ └─ HigherTaxa │ └─ HigherTaxon │ ├─ HigherTaxonName │ └─ HigherTaxonRank ├─ RecordBasis └─ Gathering ├─ DateTime │ └─ ISODateTimeBegin ├─ Agents │ └─ GatheringAgent │ └─ Person │ └─ FullName ├─ LocalityText ├─ Country │ ├─ ISO3166Code │ └─ Name └─ SiteCoordinateSets └─ SiteCoordinates └─ CoordinatesLatLong ├─ LongitudeDecimal ├─ LatitudeDecimal └─ CoordinateErrorDistanceInMeters </pre> == Commonly Used == List of further commonly mapped ABCD concepts. <pre> DataSets └─ DataSet ├─ Metadata │ └─ IPRStatements │ ├─ TermsOfUseStatements │ │ └─ TermsOfUse │ │ └─ Text │ ├─ Copyrights │ │ └─ Copyright │ │ └─ Text │ ├─ IPRDeclarations │ │ └─ IPRDeclaration │ │ └─ Text │ ├─ Licenses │ │ └─ License │ │ └─ Text │ ├─ Disclaimers │ │ └─ Disclaimer │ │ └─ Text │ ├─ Citations │ │ └─ Citation │ │ └─ Text │ └─ Acknowledgements │ └─ Acknowledgement │ └─ Text └─ Units └─ Unit ├─ DateLastEdited ├─ Identifications │ └─ Identification │ ├─ Date │ │ └─ ISODateTimeBegin │ └─ Identifiers │ └─ Identifier │ └─ IdentifierPersonName │ └─ FullName ├─ KindOfUnit ├─ CollectorsFieldNumber ├─ Sex ├─ Age ├─ Notes ├─ RecordURI ├─ MultiMediaObjects │ └─ MultiMediaObject │ ├─ FileURI │ ├─ Comment │ └─ Context ├─ SpecimenUnit │ └─ NomenclaturalTypeDesignations │ └─ NomenclaturalTypeDesignation │ ├─ TypeStatus │ ├─ TypifiedName │ │ └─ FullScientificNameString │ └─ NomenclaturalReference │ ├─ TitleCitation │ └─ CitationDetail └─ Gathering ├─ Project │ └─ ProjectTitle ├─ Biotope │ └─ Name ├─ Altitude │ └─ MeasurementOrFactAtomised │ ├─ LowerValue │ └─ UnitOfMeasurement ├─ Depth │ └─ MeasurementOrFactAtomised │ ├─ LowerValue │ └─ UnitOfMeasurement └─ NamedAreas └─ NamedArea ├─ AreaName └─ AreaClass </pre> 7d3a8963d4dc16c56ebd19972581d0ef9605a362 893 891 2016-08-03T17:25:02Z JoergHoletschek 3 /* Highly Recommended */ wikitext text/x-wiki This page lists the most commonly used ABCD 2.06 concepts. The ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a full list of ABCD concepts [http://wiki.tdwg.org/twiki/bin/view/ABCD/AbcdConcepts here]. == Mandatory == List of mandatory concepts. Without them being mapped the Provider Software will not be able to create valid ABCD document: <pre> DataSets └─ DataSet ├─ ContentContacts │ └─ ContentContact │ └─ Name ├─ TechnicalContacts │ └─ TechnicalContact │ └─ Name ├─ Metadata │ ├─ Description │ │ └─ Representation │ │ ├─ @language │ │ └─ Title │ └─ RevisionData │ └─ DateModified └─ Units └─ Unit ├─ SourceInstitutionID ├─ SourceID └─ UnitID </pre> == Highly Recommended == List of recommended ABCD concepts. If you can provide this data, please do so, they're commonly used entry points to your data. '''Note:''' If you publish your data to GBIF, you are required to specify a license in the ''Citation'' node with the ''language'' attribute and the elements ''Text'' and ''URI''. The latter should contain a link to a well-defined license, for example https://creativecommons.org/publicdomain/zero/1.0/. More details on GBIF licensing can be found [http://www.gbif.org/terms/licences here]. <pre> DataSets └─ DataSet ├─ ContentContacts │ └─ ContentContact │ ├─ Email │ └─ Address ├─ TechnicalContacts │ └─ TechnicalContact │ ├─ Email │ └─ Address ├─ Metadata │ ├─ Description │ │ └─ Representation │ │ └─ Details │ ├─ Owners │ │ └─ Owner │ │ ├─ LogoURI │ │ ├─ URIs │ │ │ └─ URL │ │ ├─ Person │ │ │ └─ FullName │ │ └─ Organisation │ │ └─ Name │ │ └─ Representation │ │ └─ Text │ └─ IPRStatements │ └─ Citations │ └─ Citation │ ├─ @language │ ├─ Text │ └─ URI └─ Units └─ Unit ├─ Identifications │ └─ Identification │ └─ Result │ ├─ @PreferredFlag │ ├─ TaxonIdentified │ │ └─ ScientificName │ │ └─ FullScientificNameString │ └─ HigherTaxa │ └─ HigherTaxon │ ├─ HigherTaxonName │ └─ HigherTaxonRank ├─ RecordBasis └─ Gathering ├─ DateTime │ └─ ISODateTimeBegin ├─ Agents │ └─ GatheringAgent │ └─ Person │ └─ FullName ├─ LocalityText ├─ Country │ ├─ ISO3166Code │ └─ Name └─ SiteCoordinateSets └─ SiteCoordinates └─ CoordinatesLatLong ├─ LongitudeDecimal ├─ LatitudeDecimal └─ CoordinateErrorDistanceInMeters </pre> == Commonly Used == List of further commonly mapped ABCD concepts. <pre> DataSets └─ DataSet ├─ Metadata │ └─ IPRStatements │ ├─ TermsOfUseStatements │ │ └─ TermsOfUse │ │ └─ Text │ ├─ Copyrights │ │ └─ Copyright │ │ └─ Text │ ├─ IPRDeclarations │ │ └─ IPRDeclaration │ │ └─ Text │ ├─ Licenses │ │ └─ License │ │ └─ Text │ ├─ Disclaimers │ │ └─ Disclaimer │ │ └─ Text │ ├─ Citations │ │ └─ Citation │ │ └─ Text │ └─ Acknowledgements │ └─ Acknowledgement │ └─ Text └─ Units └─ Unit ├─ DateLastEdited ├─ Identifications │ └─ Identification │ ├─ Date │ │ └─ ISODateTimeBegin │ └─ Identifiers │ └─ Identifier │ └─ IdentifierPersonName │ └─ FullName ├─ KindOfUnit ├─ CollectorsFieldNumber ├─ Sex ├─ Age ├─ Notes ├─ RecordURI ├─ MultiMediaObjects │ └─ MultiMediaObject │ ├─ FileURI │ ├─ Comment │ └─ Context ├─ SpecimenUnit │ └─ NomenclaturalTypeDesignations │ └─ NomenclaturalTypeDesignation │ ├─ TypeStatus │ ├─ TypifiedName │ │ └─ FullScientificNameString │ └─ NomenclaturalReference │ ├─ TitleCitation │ └─ CitationDetail └─ Gathering ├─ Project │ └─ ProjectTitle ├─ Biotope │ └─ Name ├─ Altitude │ └─ MeasurementOrFactAtomised │ ├─ LowerValue │ └─ UnitOfMeasurement ├─ Depth │ └─ MeasurementOrFactAtomised │ ├─ LowerValue │ └─ UnitOfMeasurement └─ NamedAreas └─ NamedArea ├─ AreaName └─ AreaClass </pre> b87148f0e8ced149a67a8742214642174792c9df 895 893 2016-08-04T15:18:07Z JoergHoletschek 3 /* Highly Recommended */ wikitext text/x-wiki This page lists the most commonly used ABCD 2.06 concepts. The ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki], a full list of ABCD concepts [http://wiki.tdwg.org/twiki/bin/view/ABCD/AbcdConcepts here]. == Mandatory == List of mandatory concepts. Without them being mapped the Provider Software will not be able to create valid ABCD document: <pre> DataSets └─ DataSet ├─ ContentContacts │ └─ ContentContact │ └─ Name ├─ TechnicalContacts │ └─ TechnicalContact │ └─ Name ├─ Metadata │ ├─ Description │ │ └─ Representation │ │ ├─ @language │ │ └─ Title │ └─ RevisionData │ └─ DateModified └─ Units └─ Unit ├─ SourceInstitutionID ├─ SourceID └─ UnitID </pre> == Highly Recommended == List of recommended ABCD concepts. If you can provide this data, please do so, they're commonly used entry points to your data. '''Note:''' If you publish your data to GBIF, you are required to specify a license in the ''License'' node with the ''language'' attribute and the elements ''Text'' and ''URI''. The latter should contain a link to a well-defined license, for example https://creativecommons.org/publicdomain/zero/1.0/. More details on GBIF licensing can be found [http://www.gbif.org/terms/licences here]. <pre> DataSets └─ DataSet ├─ ContentContacts │ └─ ContentContact │ ├─ Email │ └─ Address ├─ TechnicalContacts │ └─ TechnicalContact │ ├─ Email │ └─ Address ├─ Metadata │ ├─ Description │ │ └─ Representation │ │ └─ Details │ ├─ Owners │ │ └─ Owner │ │ ├─ LogoURI │ │ ├─ URIs │ │ │ └─ URL │ │ ├─ Person │ │ │ └─ FullName │ │ └─ Organisation │ │ └─ Name │ │ └─ Representation │ │ └─ Text │ └─ IPRStatements │ └─ Licenses │ └─ License │ ├─ @language │ ├─ Text │ └─ URI └─ Units └─ Unit ├─ Identifications │ └─ Identification │ └─ Result │ ├─ @PreferredFlag │ ├─ TaxonIdentified │ │ └─ ScientificName │ │ └─ FullScientificNameString │ └─ HigherTaxa │ └─ HigherTaxon │ ├─ HigherTaxonName │ └─ HigherTaxonRank ├─ RecordBasis └─ Gathering ├─ DateTime │ └─ ISODateTimeBegin ├─ Agents │ └─ GatheringAgent │ └─ Person │ └─ FullName ├─ LocalityText ├─ Country │ ├─ ISO3166Code │ └─ Name └─ SiteCoordinateSets └─ SiteCoordinates └─ CoordinatesLatLong ├─ LongitudeDecimal ├─ LatitudeDecimal └─ CoordinateErrorDistanceInMeters </pre> == Commonly Used == List of further commonly mapped ABCD concepts. <pre> DataSets └─ DataSet ├─ Metadata │ └─ IPRStatements │ ├─ TermsOfUseStatements │ │ └─ TermsOfUse │ │ └─ Text │ ├─ Copyrights │ │ └─ Copyright │ │ └─ Text │ ├─ IPRDeclarations │ │ └─ IPRDeclaration │ │ └─ Text │ ├─ Licenses │ │ └─ License │ │ └─ Text │ ├─ Disclaimers │ │ └─ Disclaimer │ │ └─ Text │ ├─ Citations │ │ └─ Citation │ │ └─ Text │ └─ Acknowledgements │ └─ Acknowledgement │ └─ Text └─ Units └─ Unit ├─ DateLastEdited ├─ Identifications │ └─ Identification │ ├─ Date │ │ └─ ISODateTimeBegin │ └─ Identifiers │ └─ Identifier │ └─ IdentifierPersonName │ └─ FullName ├─ KindOfUnit ├─ CollectorsFieldNumber ├─ Sex ├─ Age ├─ Notes ├─ RecordURI ├─ MultiMediaObjects │ └─ MultiMediaObject │ ├─ FileURI │ ├─ Comment │ └─ Context ├─ SpecimenUnit │ └─ NomenclaturalTypeDesignations │ └─ NomenclaturalTypeDesignation │ ├─ TypeStatus │ ├─ TypifiedName │ │ └─ FullScientificNameString │ └─ NomenclaturalReference │ ├─ TitleCitation │ └─ CitationDetail └─ Gathering ├─ Project │ └─ ProjectTitle ├─ Biotope │ └─ Name ├─ Altitude │ └─ MeasurementOrFactAtomised │ ├─ LowerValue │ └─ UnitOfMeasurement ├─ Depth │ └─ MeasurementOrFactAtomised │ ├─ LowerValue │ └─ UnitOfMeasurement └─ NamedAreas └─ NamedArea ├─ AreaName └─ AreaClass </pre> 454cce88a766a3af5c0d02be4a40d8ab96482472 898 895 2016-09-12T08:48:04Z JoergHoletschek 3 wikitext text/x-wiki This page lists the most commonly used ABCD 2.06 concepts. The ABCD documentation can be found in the [http://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki], a tabular list of ABCD concepts [http://terms.tdwg.org/wiki/ABCD_2_XPath_to_Short_Name_Mapping here]. == Mandatory == List of mandatory concepts. Without them being mapped the Provider Software will not be able to create valid ABCD document: <pre> DataSets └─ DataSet ├─ ContentContacts │ └─ ContentContact │ └─ Name ├─ TechnicalContacts │ └─ TechnicalContact │ └─ Name ├─ Metadata │ ├─ Description │ │ └─ Representation │ │ ├─ @language │ │ └─ Title │ └─ RevisionData │ └─ DateModified └─ Units └─ Unit ├─ SourceInstitutionID ├─ SourceID └─ UnitID </pre> == Highly Recommended == List of recommended ABCD concepts. If you can provide this data, please do so, they're commonly used entry points to your data. '''Note:''' If you publish your data to GBIF, you are required to specify a license in the ''License'' node with the ''language'' attribute and the elements ''Text'' and ''URI''. The latter should contain a link to a well-defined license, for example https://creativecommons.org/publicdomain/zero/1.0/. More details on GBIF licensing can be found [http://www.gbif.org/terms/licences here]. <pre> DataSets └─ DataSet ├─ ContentContacts │ └─ ContentContact │ ├─ Email │ └─ Address ├─ TechnicalContacts │ └─ TechnicalContact │ ├─ Email │ └─ Address ├─ Metadata │ ├─ Description │ │ └─ Representation │ │ └─ Details │ ├─ Owners │ │ └─ Owner │ │ ├─ LogoURI │ │ ├─ URIs │ │ │ └─ URL │ │ ├─ Person │ │ │ └─ FullName │ │ └─ Organisation │ │ └─ Name │ │ └─ Representation │ │ └─ Text │ └─ IPRStatements │ └─ Licenses │ └─ License │ ├─ @language │ ├─ Text │ └─ URI └─ Units └─ Unit ├─ Identifications │ └─ Identification │ └─ Result │ ├─ @PreferredFlag │ ├─ TaxonIdentified │ │ └─ ScientificName │ │ └─ FullScientificNameString │ └─ HigherTaxa │ └─ HigherTaxon │ ├─ HigherTaxonName │ └─ HigherTaxonRank ├─ RecordBasis └─ Gathering ├─ DateTime │ └─ ISODateTimeBegin ├─ Agents │ └─ GatheringAgent │ └─ Person │ └─ FullName ├─ LocalityText ├─ Country │ ├─ ISO3166Code │ └─ Name └─ SiteCoordinateSets └─ SiteCoordinates └─ CoordinatesLatLong ├─ LongitudeDecimal ├─ LatitudeDecimal └─ CoordinateErrorDistanceInMeters </pre> == Commonly Used == List of further commonly mapped ABCD concepts. <pre> DataSets └─ DataSet ├─ Metadata │ └─ IPRStatements │ ├─ TermsOfUseStatements │ │ └─ TermsOfUse │ │ └─ Text │ ├─ Copyrights │ │ └─ Copyright │ │ └─ Text │ ├─ IPRDeclarations │ │ └─ IPRDeclaration │ │ └─ Text │ ├─ Licenses │ │ └─ License │ │ └─ Text │ ├─ Disclaimers │ │ └─ Disclaimer │ │ └─ Text │ ├─ Citations │ │ └─ Citation │ │ └─ Text │ └─ Acknowledgements │ └─ Acknowledgement │ └─ Text └─ Units └─ Unit ├─ DateLastEdited ├─ Identifications │ └─ Identification │ ├─ Date │ │ └─ ISODateTimeBegin │ └─ Identifiers │ └─ Identifier │ └─ IdentifierPersonName │ └─ FullName ├─ KindOfUnit ├─ CollectorsFieldNumber ├─ Sex ├─ Age ├─ Notes ├─ RecordURI ├─ MultiMediaObjects │ └─ MultiMediaObject │ ├─ FileURI │ ├─ Comment │ └─ Context ├─ SpecimenUnit │ └─ NomenclaturalTypeDesignations │ └─ NomenclaturalTypeDesignation │ ├─ TypeStatus │ ├─ TypifiedName │ │ └─ FullScientificNameString │ └─ NomenclaturalReference │ ├─ TitleCitation │ └─ CitationDetail └─ Gathering ├─ Project │ └─ ProjectTitle ├─ Biotope │ └─ Name ├─ Altitude │ └─ MeasurementOrFactAtomised │ ├─ LowerValue │ └─ UnitOfMeasurement ├─ Depth │ └─ MeasurementOrFactAtomised │ ├─ LowerValue │ └─ UnitOfMeasurement └─ NamedAreas └─ NamedArea ├─ AreaName └─ AreaClass </pre> 3be3785e1920cf72c1882ed25ac3fad94d14e0d3 0 29 899 462 2016-09-12T08:49:14Z JoergHoletschek 3 /* Access to Biological Collection Data Schema (ABCD) */ wikitext text/x-wiki This tutorial will explain how to set up a mapping for the ABCD2.06 data schema. If you intend to map another schema (e.g. one of the ABCD extension ABCD-EFG, ABCD-DNA or HISPID), you still should take the time to read it. Large portions of the process will be identical. == Access to Biological Collection Data Schema (ABCD) == ABCD in its current version 2.06 is the data schema typically used in conjunction with BioCASe. It is a highly complex XML data schema with about 1,000 elements that is able to store almost every piece of information that can be found in a natural history specimen collection or an observation database. The complexity is due to the fact that it can be used for a wide range of collections/databases – for living and preserved specimens, observations and culture collections, for zoological, botanical, bacterial and viral collections, marine or terrestrial, for herbaria, botanic and zoological gardens. For each of these special types, ABCD features special sections where information specific for this type can be stored. Other sections of ABCD will be shared by all these types, for example for gathering site, identifier or metadata. The full ABCD documentation can be found in the http://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki], a list of commonly used ABCD elements on the Wiki page [[CommonABCD2Concepts|Common ABCD2 Concepts]]. == Creating an ABCD Mapping for a Data Source == If you created a data source from a template (e.g. ''abcdmetadata''), a schema mapping for ABCD will be already existent. In this case, you can directly jump to the next section. If you created an empty data source, the section <small>Schemas</small> in the data source configuration overview will be empty: [[File:abcdSchemaMappings.png|border]] In order to create a new schema mapping, select the desired schema from the list and click <small>Create</small>. For ABCD, choose <small>ABCD_2.06.xml</small> (be careful not to hit one of the ABCD extensions, they look quite similar). After clicking <small>Create</small>, you will be directed to the mapping editor, which will allow you to edit the mappings. == The Mapping Editor == The mapping editor allows you to add, remove and edit ABCD mappings. On top you’ll find some summary information, on the bottom you’ll see the structure of the ABCD mapping: [[File:abcdMappingEditorEmpty.png|border]] First thing we will do is to set the root table, which is the table that holds the records you want to publish (specimens or observations). That table must hold one record per object to be published, identified by a unique identifier (called ''UnitID''). This identifier doesn't need to be the primary key necessarily, as long as it is unique (if that sentence confuses you, ignore it and just keep in mind that the UnitID must be unique.) Use the drop down box labelled <small>Root table Alias</small> to select the root table of your data model. If your metadata table holds just one record (because all occurrences published share the same metadata) and if it is not linked to the root table by a foreign key, you can specify this table in the drop-down box <small>Static table Alias</small>. The Provider Software will use a natural join to connect the metadata record stored in this table to all occurrences published. If your metadata table is referenced by a foreign key in the root able, you must not declare the metadata table as a static table! So either join the metadata table using a foreign key OR declare it as a static table. '''Note:''' If you're using Access, you cannot use a static table, because Access doesn't support that. So when using Access, you always have to reference the metadata table with a foreign key. When you’ve set the root table (and static table, if you use that), press <small>Save</small>. As with the DB structure setup, this will write your changes to the configuration files, closing the tab without saving will discard your changes! Also, in case you messed something up, you can use the <small>Revert</small> button to restore the state when you’ve saved for the last time. In the mapping editor, the link <small>Overview</small> at the top of the page will take you back to the datasource configuration overview. After creating a mapping, it will appear in the section <small>Schemas</small>. The entry will show the number of mapped elements (0 for now) and the schema namespace. Pressing the trash can symbol will move it to the trash can, which means that your web service does not support this schema anymore; if you did this by mistake, you can restore it by pressing the <small>Restore</small> button. <small>Purge</small> will delete the schema mapping permanently: [[File:abcdSchemaMappings2.png|border]] You can map several schemas for a given datasource. For example, in addition to the current ABCD2 standard, you could add the deprecated version 1.2 (even though I have no idea why you should do that) or DarwinCore. Just select the appropriate schema in the drop-down list and click <small>Create</small>. Since the namespace of a schema will be used in requests, you cannot create two mappings for the same namespace. Trying this will get you an error message. == Adding Mandatory ABCD Elements == Clicking on a schema in the datasource overview page will take you again to the mapping editor. In the lower half of the page, you will see the basic structure of an ABCD document: [[File:abcdMappingEditorEmptyBottom.png|border]] <strong>Technical/Content Contact:</strong> These trees hold information on who should be contacted for technical issues or questions concerning the published database, each with name, address, email address and telephone number. <strong>Metadata:</strong> In this sub tree you’ll find all metadata fields, for example title and description of the dataset, ownership information, date last modified, IPR statements etc. <strong>Units:</strong> This part stores the "real" data you want to publish, that means all specimen or observation data. The three elements already shown in the tree are identifiers that make up the so called "Triple ID", which uniquely identifies records within primary biodiversity networks such as BioCASe and GBIF: * ''UnitID'', which is a unique identifier for each record to be published (you can use the catalogue number, for example, or an existing barcode); * ''SourceInstitutionID'', an identifier for the institution publishing the dataset (for example BGBM); * ''SourceID'', an identifier for the dataset (e.g. “Herbarium Berolinense’’). The elements already shown in the mapping editor are printed in red because they are mandatory, which means they are required for an ABCD document. Without them being mapped (to database fields that are non-empty), the Provider Software will not be able to construct valid ABCD documents and will not publish any records. So the first elements that should be mapped are the mandatory fields. To add a mapping, click the <strong>+</strong> symbol next to the field, which will open the mapping editor dialog, where you can select the database table/column you want to map the ABCD element to: [[File:abcdMappingEditorDialog.png|border]] In this case, the element ''/DataSets/DataSet/Metadata/RevisionData/DateModified'' is mapped to the column ''source_update'' of the ''metadata'' table, which has the data type ''date''. (Remember that table/column retrieval is not supported for all DBMS, so if you’re unlucky, you might need to type in table/column name manually.) <small>Save</small> will accept your settings and close the dialog, <small>Cancel</small> will discard any changes. Instead of mapping an element to a database field, you can map it to a literal. For example, you could enter the technical and content contact information directly as literals in the mapping editor dialog (assuming they’re the same for all records). However, we strongly discourage you from doing this. For one thing, this is just bad style, since it will scatter the information published over different places: Some will be retrieved from the database, some (the literals) will be loaded from the configuration files. Changing the metadata of your service later would then require access to the database and access to the BioCASe configuration – that means the configuration tool password. Be nice to the person who will do these changes and put everything neatly in a database table! The second reason is even more important: Using literals for an element will result in this element to become not searchable, which means it cannot be used in a filter of a Search or a Scan request. Even though this behaviour is discoverable with a Capabilities request (by evaluating the ''searchable'' flag), a potential user of your web service might just not think of that, so it’s a good idea to avoid that in first place. Please use literals sparsely! In the mapping editor window, you can select several source columns and several literals, even a combination of both. The Provider Software will simply concatenate the values retrieved from the database and the literals specified (in the order they’re given). But be careful: any use of concatenation or literals will make the element not searchable. A better solution is to do such manipulations earlier, in a database view, and direct the mapping editor to a single resulting database field. Use the mapping editor window to add mappings for all eight mandatory fields: Click on the <strong>+</strong> symbol next to the element, choose table/column name and the correct data type, then press <small>Save</small>. When you’re done, the page should look similar to the following image. Press <small>Save</small> in the main window to write your changes to the configuration files. [[File:abcdMappingEditorMandatoryElements.png|border]] == Initial Testing of the Data Source == Once you’ve mapped all mandatory elements, you can do the first test of your BioCASe web service. Even though you haven’t mapped any real information yet, it is a good idea to check the service and solve any problems hitherto before continuing with the mapping. Click on <small>Test mapping!</small> to open the Query Form that can be used for this purpose: [[File:abcdQueryFormEmpty.png|border]] With the Query Form you can send a BioCASe request to your web service; it will wait for the reply of the web service and display the response document. On top of the page you can see the URL of your web service. In this case it is ''<nowiki>http://localhost/biocase/pywrapper.cgi?dsa=flora</nowiki>'', ''flora'' being the name of the datasource. Make sure this is correct for your installation! Into the text box below you can enter the request XML text. Because it would be cumbersome to enter the whole BioCASe request, you can use the links below the box to load templates. For now, click on <small>ABCD2 Search</small> to fill the box with a BioCASe Search request for documents in the ABCD2 format (the one you’ve just mapped): <syntaxhighlight> <?xml version='1.0' encoding='UTF-8'?> <request xmlns='http://www.biocase.org/schemas/protocol/1.3'> <header><type>search</type></header> <search> <requestFormat>http://www.tdwg.org/schemas/abcd/2.06</requestFormat> <responseFormat start='0' limit='10'>http://www.tdwg.org/schemas/abcd/2.06</responseFormat> <filter> <like path='/DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString'>A*</like> </filter> <count>false</count> </search> </request> </syntaxhighlight> As you can see, there is a filter set on the ''ScientificName'' element. Since we haven’t mapped this element yet, this won’t work. So let’s remove the filter (all lines between <filter> and </filter>, including the ''filter'' tags), leaving the text box like this: [[File:abcdQueryFormFilled.png|border]] Pressing <small>Submit</small> will send the request to the web service and – hopefully – display the response document with the correct ABCD records. For the mapping we’ve just created, it should look similar to the following (use the "–" symbol next to the trees TechnicalContact, ContentContact and Metadata to collapse them): [[File:abcdResponseMandatoryElements.png|border]] If you get an error, please check out the [[Debugging]] tutorial on how to debug a web service. You should continue the mapping process only after your web service returns results for the basal mapping we’ve just created. == Adding additional ABCD Elements == Once you’ve successfully managed to map the mandatory ABCD elements ("successfully" means the web service returns ABCD documents), you can go on with the mapping process. This is usually an iterative process: You add a handful of elements (for example, country, country code, locality text and coordinates for the gathering site), then you use the Query Form to see if the information ends up where it is supposed to do. This is the reason the Query Form opens up in a separate tab (or window, of you configured your web browser so): In one tab, you use the mapping editor to add new elements; after pressing <small>Save</small> you go to the Query Form tab and see what the response documents look like. Redo this until you’ve mapped all information you want to publish. In order to map new ABCD concepts, tick the check box labelled <small>Show all concepts</small> and press <small>Refresh</small>. The mapping editor will then display the whole ABCD tree, which looks admittedly a bit intimidating. In order to find the concept you’re looking for, use the search function of your browser. If you want to map the country name of the gathering site, for example, hit <small>Ctrl + F</small> (for Firefox, Microsoft IE and Opera) and type in the name of the ABCD element, namely ''country''. The browser will jump to the ''Country'' tree of the ''Gathering'' element, where you’ll find the elements for country name (''Name''), ISO country code (''ISO3166Code'') and the country name in the country’s language (''NameDerived''): [[File:abcdMappingEditorSearch.png|500px]] Continue the sequence mapping new elements/testing until you’ve mapped all information you want to publish. You can have a look at the list of recommend and commonly used ABCD elements on the [[CommonABCD2Concepts|Common ABCD2 Concepts]] page. The full ABCD documentation can be found in the [http://wiki.tdwg.org/ABCD ABCD Wiki]. The following page will show you a sample response ABCD document for a web service with 77 concepts mapped. Of course your documents will look different, because you will probably use different ABCD elements, but still it’s good to get an idea: [[SampleABCDDocument]]. 0620093a469d52ee1fc0713793ff7d17440e74cd 900 899 2016-09-12T08:49:41Z JoergHoletschek 3 /* Adding additional ABCD Elements */ wikitext text/x-wiki This tutorial will explain how to set up a mapping for the ABCD2.06 data schema. If you intend to map another schema (e.g. one of the ABCD extension ABCD-EFG, ABCD-DNA or HISPID), you still should take the time to read it. Large portions of the process will be identical. == Access to Biological Collection Data Schema (ABCD) == ABCD in its current version 2.06 is the data schema typically used in conjunction with BioCASe. It is a highly complex XML data schema with about 1,000 elements that is able to store almost every piece of information that can be found in a natural history specimen collection or an observation database. The complexity is due to the fact that it can be used for a wide range of collections/databases – for living and preserved specimens, observations and culture collections, for zoological, botanical, bacterial and viral collections, marine or terrestrial, for herbaria, botanic and zoological gardens. For each of these special types, ABCD features special sections where information specific for this type can be stored. Other sections of ABCD will be shared by all these types, for example for gathering site, identifier or metadata. The full ABCD documentation can be found in the http://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki], a list of commonly used ABCD elements on the Wiki page [[CommonABCD2Concepts|Common ABCD2 Concepts]]. == Creating an ABCD Mapping for a Data Source == If you created a data source from a template (e.g. ''abcdmetadata''), a schema mapping for ABCD will be already existent. In this case, you can directly jump to the next section. If you created an empty data source, the section <small>Schemas</small> in the data source configuration overview will be empty: [[File:abcdSchemaMappings.png|border]] In order to create a new schema mapping, select the desired schema from the list and click <small>Create</small>. For ABCD, choose <small>ABCD_2.06.xml</small> (be careful not to hit one of the ABCD extensions, they look quite similar). After clicking <small>Create</small>, you will be directed to the mapping editor, which will allow you to edit the mappings. == The Mapping Editor == The mapping editor allows you to add, remove and edit ABCD mappings. On top you’ll find some summary information, on the bottom you’ll see the structure of the ABCD mapping: [[File:abcdMappingEditorEmpty.png|border]] First thing we will do is to set the root table, which is the table that holds the records you want to publish (specimens or observations). That table must hold one record per object to be published, identified by a unique identifier (called ''UnitID''). This identifier doesn't need to be the primary key necessarily, as long as it is unique (if that sentence confuses you, ignore it and just keep in mind that the UnitID must be unique.) Use the drop down box labelled <small>Root table Alias</small> to select the root table of your data model. If your metadata table holds just one record (because all occurrences published share the same metadata) and if it is not linked to the root table by a foreign key, you can specify this table in the drop-down box <small>Static table Alias</small>. The Provider Software will use a natural join to connect the metadata record stored in this table to all occurrences published. If your metadata table is referenced by a foreign key in the root able, you must not declare the metadata table as a static table! So either join the metadata table using a foreign key OR declare it as a static table. '''Note:''' If you're using Access, you cannot use a static table, because Access doesn't support that. So when using Access, you always have to reference the metadata table with a foreign key. When you’ve set the root table (and static table, if you use that), press <small>Save</small>. As with the DB structure setup, this will write your changes to the configuration files, closing the tab without saving will discard your changes! Also, in case you messed something up, you can use the <small>Revert</small> button to restore the state when you’ve saved for the last time. In the mapping editor, the link <small>Overview</small> at the top of the page will take you back to the datasource configuration overview. After creating a mapping, it will appear in the section <small>Schemas</small>. The entry will show the number of mapped elements (0 for now) and the schema namespace. Pressing the trash can symbol will move it to the trash can, which means that your web service does not support this schema anymore; if you did this by mistake, you can restore it by pressing the <small>Restore</small> button. <small>Purge</small> will delete the schema mapping permanently: [[File:abcdSchemaMappings2.png|border]] You can map several schemas for a given datasource. For example, in addition to the current ABCD2 standard, you could add the deprecated version 1.2 (even though I have no idea why you should do that) or DarwinCore. Just select the appropriate schema in the drop-down list and click <small>Create</small>. Since the namespace of a schema will be used in requests, you cannot create two mappings for the same namespace. Trying this will get you an error message. == Adding Mandatory ABCD Elements == Clicking on a schema in the datasource overview page will take you again to the mapping editor. In the lower half of the page, you will see the basic structure of an ABCD document: [[File:abcdMappingEditorEmptyBottom.png|border]] <strong>Technical/Content Contact:</strong> These trees hold information on who should be contacted for technical issues or questions concerning the published database, each with name, address, email address and telephone number. <strong>Metadata:</strong> In this sub tree you’ll find all metadata fields, for example title and description of the dataset, ownership information, date last modified, IPR statements etc. <strong>Units:</strong> This part stores the "real" data you want to publish, that means all specimen or observation data. The three elements already shown in the tree are identifiers that make up the so called "Triple ID", which uniquely identifies records within primary biodiversity networks such as BioCASe and GBIF: * ''UnitID'', which is a unique identifier for each record to be published (you can use the catalogue number, for example, or an existing barcode); * ''SourceInstitutionID'', an identifier for the institution publishing the dataset (for example BGBM); * ''SourceID'', an identifier for the dataset (e.g. “Herbarium Berolinense’’). The elements already shown in the mapping editor are printed in red because they are mandatory, which means they are required for an ABCD document. Without them being mapped (to database fields that are non-empty), the Provider Software will not be able to construct valid ABCD documents and will not publish any records. So the first elements that should be mapped are the mandatory fields. To add a mapping, click the <strong>+</strong> symbol next to the field, which will open the mapping editor dialog, where you can select the database table/column you want to map the ABCD element to: [[File:abcdMappingEditorDialog.png|border]] In this case, the element ''/DataSets/DataSet/Metadata/RevisionData/DateModified'' is mapped to the column ''source_update'' of the ''metadata'' table, which has the data type ''date''. (Remember that table/column retrieval is not supported for all DBMS, so if you’re unlucky, you might need to type in table/column name manually.) <small>Save</small> will accept your settings and close the dialog, <small>Cancel</small> will discard any changes. Instead of mapping an element to a database field, you can map it to a literal. For example, you could enter the technical and content contact information directly as literals in the mapping editor dialog (assuming they’re the same for all records). However, we strongly discourage you from doing this. For one thing, this is just bad style, since it will scatter the information published over different places: Some will be retrieved from the database, some (the literals) will be loaded from the configuration files. Changing the metadata of your service later would then require access to the database and access to the BioCASe configuration – that means the configuration tool password. Be nice to the person who will do these changes and put everything neatly in a database table! The second reason is even more important: Using literals for an element will result in this element to become not searchable, which means it cannot be used in a filter of a Search or a Scan request. Even though this behaviour is discoverable with a Capabilities request (by evaluating the ''searchable'' flag), a potential user of your web service might just not think of that, so it’s a good idea to avoid that in first place. Please use literals sparsely! In the mapping editor window, you can select several source columns and several literals, even a combination of both. The Provider Software will simply concatenate the values retrieved from the database and the literals specified (in the order they’re given). But be careful: any use of concatenation or literals will make the element not searchable. A better solution is to do such manipulations earlier, in a database view, and direct the mapping editor to a single resulting database field. Use the mapping editor window to add mappings for all eight mandatory fields: Click on the <strong>+</strong> symbol next to the element, choose table/column name and the correct data type, then press <small>Save</small>. When you’re done, the page should look similar to the following image. Press <small>Save</small> in the main window to write your changes to the configuration files. [[File:abcdMappingEditorMandatoryElements.png|border]] == Initial Testing of the Data Source == Once you’ve mapped all mandatory elements, you can do the first test of your BioCASe web service. Even though you haven’t mapped any real information yet, it is a good idea to check the service and solve any problems hitherto before continuing with the mapping. Click on <small>Test mapping!</small> to open the Query Form that can be used for this purpose: [[File:abcdQueryFormEmpty.png|border]] With the Query Form you can send a BioCASe request to your web service; it will wait for the reply of the web service and display the response document. On top of the page you can see the URL of your web service. In this case it is ''<nowiki>http://localhost/biocase/pywrapper.cgi?dsa=flora</nowiki>'', ''flora'' being the name of the datasource. Make sure this is correct for your installation! Into the text box below you can enter the request XML text. Because it would be cumbersome to enter the whole BioCASe request, you can use the links below the box to load templates. For now, click on <small>ABCD2 Search</small> to fill the box with a BioCASe Search request for documents in the ABCD2 format (the one you’ve just mapped): <syntaxhighlight> <?xml version='1.0' encoding='UTF-8'?> <request xmlns='http://www.biocase.org/schemas/protocol/1.3'> <header><type>search</type></header> <search> <requestFormat>http://www.tdwg.org/schemas/abcd/2.06</requestFormat> <responseFormat start='0' limit='10'>http://www.tdwg.org/schemas/abcd/2.06</responseFormat> <filter> <like path='/DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString'>A*</like> </filter> <count>false</count> </search> </request> </syntaxhighlight> As you can see, there is a filter set on the ''ScientificName'' element. Since we haven’t mapped this element yet, this won’t work. So let’s remove the filter (all lines between <filter> and </filter>, including the ''filter'' tags), leaving the text box like this: [[File:abcdQueryFormFilled.png|border]] Pressing <small>Submit</small> will send the request to the web service and – hopefully – display the response document with the correct ABCD records. For the mapping we’ve just created, it should look similar to the following (use the "–" symbol next to the trees TechnicalContact, ContentContact and Metadata to collapse them): [[File:abcdResponseMandatoryElements.png|border]] If you get an error, please check out the [[Debugging]] tutorial on how to debug a web service. You should continue the mapping process only after your web service returns results for the basal mapping we’ve just created. == Adding additional ABCD Elements == Once you’ve successfully managed to map the mandatory ABCD elements ("successfully" means the web service returns ABCD documents), you can go on with the mapping process. This is usually an iterative process: You add a handful of elements (for example, country, country code, locality text and coordinates for the gathering site), then you use the Query Form to see if the information ends up where it is supposed to do. This is the reason the Query Form opens up in a separate tab (or window, of you configured your web browser so): In one tab, you use the mapping editor to add new elements; after pressing <small>Save</small> you go to the Query Form tab and see what the response documents look like. Redo this until you’ve mapped all information you want to publish. In order to map new ABCD concepts, tick the check box labelled <small>Show all concepts</small> and press <small>Refresh</small>. The mapping editor will then display the whole ABCD tree, which looks admittedly a bit intimidating. In order to find the concept you’re looking for, use the search function of your browser. If you want to map the country name of the gathering site, for example, hit <small>Ctrl + F</small> (for Firefox, Microsoft IE and Opera) and type in the name of the ABCD element, namely ''country''. The browser will jump to the ''Country'' tree of the ''Gathering'' element, where you’ll find the elements for country name (''Name''), ISO country code (''ISO3166Code'') and the country name in the country’s language (''NameDerived''): [[File:abcdMappingEditorSearch.png|500px]] Continue the sequence mapping new elements/testing until you’ve mapped all information you want to publish. You can have a look at the list of recommend and commonly used ABCD elements on the [[CommonABCD2Concepts|Common ABCD2 Concepts]] page. The full ABCD documentation can be found in the [http://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. The following page will show you a sample response ABCD document for a web service with 77 concepts mapped. Of course your documents will look different, because you will probably use different ABCD elements, but still it’s good to get an idea: [[SampleABCDDocument]]. e292a2d7d271ac0d30e1cc717a1e416d4b324270 0 80 915 842 2017-11-01T09:57:51Z JoergHoletschek 3 /* Discovery of Archives */ wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later.<br>If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousand or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The OpenUp! natural history aggregator, for example, can use this method for fast harvesting of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==Managing Archives== The section <small>Existing Archives</small> allows you download, remove and convert archives or view the log of the archiving process: [[File:xmlArchivesNew.png|border|775px]] The archives are grouped by schema, with DarwinCore archives making up an additional group. You can use the minus or plus symbol next to the group header to expand or collapse all files of that group; for web services publishing dozens of datasets, this might be handy to reduce scrolling on the page. The <small>Log </small> link in the group header will open the log file for the respective archiving process in a separate tab. For each archive file, the number of records stored in this file, the date of creation and the file size are listed. XML archives may have a number printed in the <small>Dropped</small> column, meaning some records have been dropped due to missing mandatory data elements. This doesn't mean the archive is invalid, but you should review the mapping or your source data to find the reasons for these lost records. Clicking the archive name will download the archive file, so it can be saved to disk or opened with an archive viewing tool. Pressing the trash symbol at the very right will delete the archive; if you use the trash symbol in the group header, all archives of this schema/type will be removed. For ABCD 2.06 and HISPID 5 XML archives, you will see an additional <small>DwC</small> link next to the trash symbol. This allows you to selectively convert a single dataset from XML to DarwinCore format. In contrast to using the <small>Create DwC Archive</small> button described below, this will spare the XML archiving of the whole web service and just create a DarwinCore archive for the selected dataset, which will be significantly faster. If you use the link in the group header, all archives of the schema will be converted (again without triggering XML archiving first). ==XML Archives== If you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. The section <small>Create/Update Archives</small> allows you to create new XML archive(s) for each schema mapped. Simply select the schema and press <small>Create XML Archive</small>. As mentioned above, archiving ABCD 2.06 or HISPID 5 is dataset-aware, resulting in one XML archive per dataset published. For all other schemas, a single archive file will be created. The paging size specifies how many records will be bundled into one XML document inside the archive; the default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgressNew.png|border]] You can always cancel the archiving process by pressing the <small>Cancel</small> link in the Status/Messages box. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Existing Archives</small> list, or, if it’s still being processed, as <small>Processing</small> in the Status/Messages box. If the process failed for some reason (or you decided to cancel it), a new box <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create XML archives for one schema at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. DarwinCore archives store only one dataset; so as with ABCD 2.06 and HISPID 5 XML archives, DwC-archiving a web service might result in several DarwinCore archives. DarwinCore archives are not created directly, but from the respective ABCD/HISPID archives. So DwC-archiving a web service will first create ACBD archives, which will be converted into the respective DwC archives subsequently. ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the library test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive before, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the archive name listed under ''Existing Archives''. The XML archiving step and the parameter <small>Paging size</small> are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgressNew.png|border]] You can cancel the transformation process by clicking the <small>Cancel</small> link in the Status/Messages box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from the ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: Conversion finished.'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. This parameter corresponds to the paging size that can be set on the archiving page. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. == Discovery of Archives == Archives available for a given data source can be discovered through the so-called dataset inventory. To get this, just append ''&inventory=1'' to the access URL of the data source, for example http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar&inventory=1. The dataset inventory response looks roughly like this (diagnostic messages have been removed for brevity): <syntaxhighlight> <dsi:inventory xsi:schemaLocation="http://www.biocase.org/schemas/dsi/1.0 http://www.bgbm.org/biodivinf/schema/dsi_1_0.xsd"> <!--XML generated by BioCASE PyWrapper software version 3.5.3. Made in Berlin.--> <dsi:status>OK</dsi:status> <dsi:created>2015-06-25T15:22:51.805000</dsi:created> <dsi:service_url>http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=BoBO</dsi:service_url> <dsi:datasets> <dsi:dataset> <dsi:title>BoBO - Botanic Garden and Botanical Museum Berlin-Dahlem Observations</dsi:title> <dsi:id>BoBO - Botanic Garden and Botanical Museum Berlin-Dahlem Observations</dsi:id> <dsi:archives> <dsi:archive filesize="143137" modified="2015-05-06T16:06:19.346807" namespace="http://www.tdwg.org/schemas/abcd/2.06" rcount="823">http://ww3.bgbm.org/biocase/downloads/BoBO/BoBO%20-%20Botanic%20Garden%20and%20Botanical%20Museum%20Berlin-Dahlem%20Observations.ABCD_2.06.zip</dsi:archive> <dsi:archive filesize="128502" modified="2015-05-06T16:06:34.768532" rcount="823" rowType="http://rs.tdwg.org/dwc/terms/Occurrence">http://ww3.bgbm.org/biocase/downloads/BoBO/BoBO%20-%20Botanic%20Garden%20and%20Botanical%20Museum%20Berlin-Dahlem%20Observations.DwCA.zip</dsi:archive> </dsi:archives> </dsi:dataset> </dsi:datasets> <dsi:diagnostics> ... </dsi:diagnostics> </dsi:inventory> </syntaxhighlight> The dataset inventory will list all datasets published by a given data source, each with its title and ID. If archives exist for this dataset, they will be listed in ''archives'', each ''archive'' element storing the URL for the respective file. For XML archives, the attribute ''namespace'' will store the namespace of the schema, for DarwinCore archives there will an attribute ''rowType'' with the value <nowiki>http://rs.tdwg.org/dwc/terms/Occurrence</nowiki>. The attribute ''rcount'' specifies the number of records stored in the archive file, ''filesize'' and ''modified'' are hopefully self-explanatory. You can also view the dataset inventory using the manual QueryForm: Scroll down to the section ''Configuration Info'' and click on the link ''Inventory''. After a second, it will appear in the text box below. cdcdef510ba413c93a92ec2c81d64c217e469444 916 915 2017-11-01T10:01:03Z JoergHoletschek 3 /* Discovery of Archives */ wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later.<br>If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousand or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The OpenUp! natural history aggregator, for example, can use this method for fast harvesting of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==Managing Archives== The section <small>Existing Archives</small> allows you download, remove and convert archives or view the log of the archiving process: [[File:xmlArchivesNew.png|border|775px]] The archives are grouped by schema, with DarwinCore archives making up an additional group. You can use the minus or plus symbol next to the group header to expand or collapse all files of that group; for web services publishing dozens of datasets, this might be handy to reduce scrolling on the page. The <small>Log </small> link in the group header will open the log file for the respective archiving process in a separate tab. For each archive file, the number of records stored in this file, the date of creation and the file size are listed. XML archives may have a number printed in the <small>Dropped</small> column, meaning some records have been dropped due to missing mandatory data elements. This doesn't mean the archive is invalid, but you should review the mapping or your source data to find the reasons for these lost records. Clicking the archive name will download the archive file, so it can be saved to disk or opened with an archive viewing tool. Pressing the trash symbol at the very right will delete the archive; if you use the trash symbol in the group header, all archives of this schema/type will be removed. For ABCD 2.06 and HISPID 5 XML archives, you will see an additional <small>DwC</small> link next to the trash symbol. This allows you to selectively convert a single dataset from XML to DarwinCore format. In contrast to using the <small>Create DwC Archive</small> button described below, this will spare the XML archiving of the whole web service and just create a DarwinCore archive for the selected dataset, which will be significantly faster. If you use the link in the group header, all archives of the schema will be converted (again without triggering XML archiving first). ==XML Archives== If you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. The section <small>Create/Update Archives</small> allows you to create new XML archive(s) for each schema mapped. Simply select the schema and press <small>Create XML Archive</small>. As mentioned above, archiving ABCD 2.06 or HISPID 5 is dataset-aware, resulting in one XML archive per dataset published. For all other schemas, a single archive file will be created. The paging size specifies how many records will be bundled into one XML document inside the archive; the default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgressNew.png|border]] You can always cancel the archiving process by pressing the <small>Cancel</small> link in the Status/Messages box. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Existing Archives</small> list, or, if it’s still being processed, as <small>Processing</small> in the Status/Messages box. If the process failed for some reason (or you decided to cancel it), a new box <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create XML archives for one schema at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. DarwinCore archives store only one dataset; so as with ABCD 2.06 and HISPID 5 XML archives, DwC-archiving a web service might result in several DarwinCore archives. DarwinCore archives are not created directly, but from the respective ABCD/HISPID archives. So DwC-archiving a web service will first create ACBD archives, which will be converted into the respective DwC archives subsequently. ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the library test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive before, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the archive name listed under ''Existing Archives''. The XML archiving step and the parameter <small>Paging size</small> are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgressNew.png|border]] You can cancel the transformation process by clicking the <small>Cancel</small> link in the Status/Messages box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from the ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: Conversion finished.'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. This parameter corresponds to the paging size that can be set on the archiving page. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. == Discovery of Archives == Archives available for a given data source can be discovered through the so-called dataset inventory. To get this, just append ''&inventory=1'' to the access URL of the data source, for example http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar&inventory=1. The dataset inventory response looks roughly like this (diagnostic messages have been removed for brevity): <syntaxhighlight> <dsi:inventory xmlns:dsi="http://www.biocase.org/schemas/dsi/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.biocase.org/schemas/dsi/1.0 http://www.bgbm.org/biodivinf/schema/dsi_1_0.xsd"> <!--XML generated by BioCASE PyWrapper software version 3.6.4. Made in Berlin.--> <dsi:status>OK</dsi:status> <dsi:created>2017-11-01T10:58:38.109000</dsi:created> <dsi:service_url>http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar</dsi:service_url> <dsi:datasets> <dsi:dataset> <dsi:title>Herbarium Berolinense</dsi:title> <dsi:id>Herbarium Berolinense</dsi:id> <dsi:archives> <dsi:archive filesize="20187103" modified="2017-04-20T12:59:33.810767" namespace="http://www.tdwg.org/schemas/abcd/2.06" rcount="206223">http://ww3.bgbm.org/biocase/downloads/Herbar/Herbarium%20Berolinense.ABCD_2.06.zip</dsi:archive> <dsi:archive filesize="12373841" modified="2017-04-20T13:16:03.772798" rcount="206223" rowType="http://rs.tdwg.org/dwc/terms/Occurrence">http://ww3.bgbm.org/biocase/downloads/Herbar/Herbarium%20Berolinense.DwCA.zip</dsi:archive> </dsi:archives> </dsi:dataset> </dsi:datasets> <dsi:diagnostics> ... </dsi:diagnostics> </dsi:inventory> </syntaxhighlight> The dataset inventory will list all datasets published by a given data source, each with its title and ID. If archives exist for this dataset, they will be listed in ''archives'', each ''archive'' element storing the URL for the respective file. For XML archives, the attribute ''namespace'' will store the namespace of the schema, for DarwinCore archives there will an attribute ''rowType'' with the value <nowiki>http://rs.tdwg.org/dwc/terms/Occurrence</nowiki>. The attribute ''rcount'' specifies the number of records stored in the archive file, ''filesize'' and ''modified'' are hopefully self-explanatory. You can also view the dataset inventory using the manual QueryForm: Scroll down to the section ''Configuration Info'' and click on the link ''Inventory''. After a second, it will appear in the text box below. bf5dbf0cae4e98eb497fed70b0c82d20ce7997b5 917 916 2017-11-01T10:09:41Z JoergHoletschek 3 /* Discovery of Archives */ wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later.<br>If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousand or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The OpenUp! natural history aggregator, for example, can use this method for fast harvesting of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==Managing Archives== The section <small>Existing Archives</small> allows you download, remove and convert archives or view the log of the archiving process: [[File:xmlArchivesNew.png|border|775px]] The archives are grouped by schema, with DarwinCore archives making up an additional group. You can use the minus or plus symbol next to the group header to expand or collapse all files of that group; for web services publishing dozens of datasets, this might be handy to reduce scrolling on the page. The <small>Log </small> link in the group header will open the log file for the respective archiving process in a separate tab. For each archive file, the number of records stored in this file, the date of creation and the file size are listed. XML archives may have a number printed in the <small>Dropped</small> column, meaning some records have been dropped due to missing mandatory data elements. This doesn't mean the archive is invalid, but you should review the mapping or your source data to find the reasons for these lost records. Clicking the archive name will download the archive file, so it can be saved to disk or opened with an archive viewing tool. Pressing the trash symbol at the very right will delete the archive; if you use the trash symbol in the group header, all archives of this schema/type will be removed. For ABCD 2.06 and HISPID 5 XML archives, you will see an additional <small>DwC</small> link next to the trash symbol. This allows you to selectively convert a single dataset from XML to DarwinCore format. In contrast to using the <small>Create DwC Archive</small> button described below, this will spare the XML archiving of the whole web service and just create a DarwinCore archive for the selected dataset, which will be significantly faster. If you use the link in the group header, all archives of the schema will be converted (again without triggering XML archiving first). ==XML Archives== If you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. The section <small>Create/Update Archives</small> allows you to create new XML archive(s) for each schema mapped. Simply select the schema and press <small>Create XML Archive</small>. As mentioned above, archiving ABCD 2.06 or HISPID 5 is dataset-aware, resulting in one XML archive per dataset published. For all other schemas, a single archive file will be created. The paging size specifies how many records will be bundled into one XML document inside the archive; the default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgressNew.png|border]] You can always cancel the archiving process by pressing the <small>Cancel</small> link in the Status/Messages box. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Existing Archives</small> list, or, if it’s still being processed, as <small>Processing</small> in the Status/Messages box. If the process failed for some reason (or you decided to cancel it), a new box <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create XML archives for one schema at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. DarwinCore archives store only one dataset; so as with ABCD 2.06 and HISPID 5 XML archives, DwC-archiving a web service might result in several DarwinCore archives. DarwinCore archives are not created directly, but from the respective ABCD/HISPID archives. So DwC-archiving a web service will first create ACBD archives, which will be converted into the respective DwC archives subsequently. ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the library test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive before, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the archive name listed under ''Existing Archives''. The XML archiving step and the parameter <small>Paging size</small> are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgressNew.png|border]] You can cancel the transformation process by clicking the <small>Cancel</small> link in the Status/Messages box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from the ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: Conversion finished.'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. This parameter corresponds to the paging size that can be set on the archiving page. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. == Discovery of Archives == Archives available for a given data source can be discovered through the so-called dataset inventory. To get this, just append ''&inventory=1'' to the access URL of the data source, for example http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar&inventory=1. The dataset inventory response looks roughly like this (diagnostic messages have been removed for brevity): <syntaxhighlight> <dsi:inventory xmlns:dsi="http://www.biocase.org/schemas/dsi/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.biocase.org/schemas/dsi/1.0 http://www.bgbm.org/biodivinf/schema/dsi_1_0.xsd"> <!--XML generated by BioCASE PyWrapper software version 3.6.4. Made in Berlin.--> <dsi:status>OK</dsi:status> <dsi:created>2017-11-01T10:58:38.109000</dsi:created> <dsi:service_url>http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar</dsi:service_url> <dsi:datasets> <dsi:dataset> <dsi:title>Herbarium Berolinense</dsi:title> <dsi:id>Herbarium Berolinense</dsi:id> <dsi:archives> <dsi:archive filesize="20187103" modified="2017-04-20T12:59:33.810767" namespace="http://www.tdwg.org/schemas/abcd/2.06" rcount="206223">http://ww3.bgbm.org/biocase/downloads/Herbar/Herbarium%20Berolinense.ABCD_2.06.zip</dsi:archive> <dsi:archive filesize="12373841" modified="2017-04-20T13:16:03.772798" rcount="206223" rowType="http://rs.tdwg.org/dwc/terms/Occurrence">http://ww3.bgbm.org/biocase/downloads/Herbar/Herbarium%20Berolinense.DwCA.zip</dsi:archive> </dsi:archives> </dsi:dataset> </dsi:datasets> <dsi:diagnostics> ... </dsi:diagnostics> </dsi:inventory> </syntaxhighlight> The dataset inventory will list all datasets published by a given data source, each with its title and ID. If archives exist for this dataset, they will be listed in ''archives'', each ''archive'' element storing the URL for the respective file. For XML archives, the attribute ''namespace'' will store the namespace of the schema, for DarwinCore archives there will an attribute ''rowType'' with the value <nowiki>http://rs.tdwg.org/dwc/terms/Occurrence</nowiki>. The attribute ''rcount'' specifies the number of records stored in the archive file, ''filesize'' and ''modified'' are hopefully self-explanatory. You can also view the dataset inventory using the manual QueryForm: Scroll down to the section ''Configuration Info'' and click on the link ''Inventory''. After a second, it will appear in the text box below. 35fc896c4e15f498105bc4a0703db251a69a3328 0 23 921 920 2018-03-12T16:47:45Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Fixes ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 51dc483f2e040be8b6f3b41b201c9f13eaa2f0c0 922 921 2018-03-19T15:25:50Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Fixes ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool fdd605846e2cf5a6cbeccec2f95f2d1173e37b09 926 922 2018-04-18T09:07:21Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Fixes ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 685584462f42c2cc94000f68176b07d3fbbed7ad 927 926 2018-04-18T09:12:55Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default values for bpsPath (in adjustpath.py) and graphvizPath (config.ini) set to Debian defaults for easier dockerisation. * Fixes ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 883079dd0f0467e598e3b933ce56037f3be9f0ee 928 927 2018-04-18T09:37:09Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Fixes ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 1ec034bf7a96247344d20b8496a096694984f5f3 929 928 2018-04-23T09:28:49Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fixes ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 73c361f588ce8b49907923c0d234e2fc5f2d3d76 930 929 2018-04-27T08:02:40Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * ODBC modules now use pyodbc or ceODBC, whichever is installed. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fixes ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool e280ba5305853e501289949f4debbf16045af6c8 931 930 2018-05-25T11:14:29Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fixes ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 88a4019f4c95cb207238f898497a3a406161f7f0 967 931 2018-05-30T08:40:02Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 818b0a702301cd71c4b6794cc0899d369310714b 968 967 2018-05-30T09:23:45Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool f7a627b3155e2870067856154fbb75678eb82da4 969 968 2018-05-30T09:25:26Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully dunctional demo data source included (uses included Sqlite database). * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool d518cf901f42ff9e120d9a9408b3e1b75d98e07f 970 969 2018-06-12T08:53:28Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully dunctional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 02199eb8147725174e02d2c3c6e089322c5bc78f 0 40 923 282 2018-04-10T15:15:59Z DavidFichtmueller 4 xml errors removed wikitext text/x-wiki This page shows a sample response ABCD2 document for a web service with 77 concepts mapped. For easier viewing, you can save the contents of the box into a file with the extension .xml and open that in Firefix or MS Internet Explorer (don't use Safari). You will then be able to collapse the different XML sub trees. <syntaxhighlight lang="xml"> <?xml version='1.0' encoding='UTF-8'?> <biocase:response xmlns:abcd="http://www.tdwg.org/schemas/abcd/2.06" xmlns:biocase="http://www.biocase.org/schemas/protocol/1.3" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.tdwg.org/schemas/abcd/2.06 http://www.bgbm.org/TDWG/CODATA/Schema/ABCD_2.06/ABCD_2.06.XSD http://www.biocase.org/schemas/protocol/1.3 http://www.bgbm.org/biodivinf/schema/protocol_1_31.xsd"> <!-- XML generated by BioCASE PyWrapper software version 2.6.0. Made in Berlin. --> <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> <biocase:content recordCount="10" recordDropped="0" recordStart="0" totalSearchHits="11"> <abcd:DataSets> <abcd:DataSet> <abcd:TechnicalContacts> <abcd:TechnicalContact> <abcd:Name>Botanic Garden and Botanical Museum Berlin-Dahlem</abcd:Name> <abcd:Email>...@...</abcd:Email> <abcd:Address>Königin-Luise-Str. 6-8, D-14195 Berlin</abcd:Address> </abcd:TechnicalContact> </abcd:TechnicalContacts> <abcd:ContentContacts> <abcd:ContentContact> <abcd:Name>Dr. ...</abcd:Name> <abcd:Email>...@...</abcd:Email> <abcd:Phone>+49 ...</abcd:Phone> <abcd:Address>..., D-22609 Hamburg</abcd:Address> </abcd:ContentContact> </abcd:ContentContacts> <abcd:Metadata> <abcd:Description> <abcd:Representation language="EN"> <abcd:Title>Desmidiaceae Engels</abcd:Title> <abcd:Details>A new record of reference material of the Desmidiaceae of Germany was compiled, using the databank “Specify”. Aim of the project was to summarize and digitalize the data of the dried and otherwise preserved specimens stored in the German herbaria as well as of the living strains, cultured in culture collections. Many of the references are completed by figures of the labels or by scanned micrographs from preparations for the light- or scanning microscopes.</abcd:Details> <abcd:URI>http://www.gbif.de/botanik/projekte/desmidiaceae_hamburg_ffm/index_html</abcd:URI> </abcd:Representation> </abcd:Description> <abcd:IconURI>http://ww3.bgbm.org/providerResources/BzF.jpg</abcd:IconURI> <abcd:RevisionData> <abcd:Creators>Desmidiaceae Engels (BzF)</abcd:Creators> <abcd:DateModified>2008-01-23T00:00:00</abcd:DateModified> </abcd:RevisionData> <abcd:Owners> <abcd:Owner> <abcd:Organisation> <abcd:Name> <abcd:Representation language="EN"> <abcd:Text>Biozentrum Klein-Flottbek, Universität Hamburg</abcd:Text> <abcd:Abbreviation>BzF</abcd:Abbreviation> </abcd:Representation> </abcd:Name> </abcd:Organisation> </abcd:Owner> </abcd:Owners> <abcd:IPRStatements> <abcd:IPRDeclarations> <abcd:IPRDeclaration language="EN"> <abcd:Text>The Intellectual Property Rights are held by the legal owner or, in case of living persons, by the collector or determinator.</abcd:Text> </abcd:IPRDeclaration> </abcd:IPRDeclarations> <abcd:Copyrights> <abcd:Copyright language="EN"> <abcd:Text>No part of this data base may be copied or reproduced without written permission from the legal owner.</abcd:Text> </abcd:Copyright> </abcd:Copyrights> <abcd:TermsOfUseStatements> <abcd:TermsOfUse language="EN"> <abcd:Text>The use of the data is allowed only for non-profit scientific use and for non-profit nature conservation purpose. The database or part of it may only be used or copied by the written permission from the legal owner. If used for publication, we ask for a copy or an off-print.</abcd:Text> </abcd:TermsOfUse> </abcd:TermsOfUseStatements> <abcd:Disclaimers> <abcd:Disclaimer language="EN"> <abcd:Text>No responsibility is accepted for the accuracy of the information in this database.</abcd:Text> </abcd:Disclaimer> </abcd:Disclaimers> <abcd:Acknowledgements> <abcd:Acknowledgement language="EN"> <abcd:Text>The authors are much indebted to the Scientists of the Informatics Biodiversity Research Center der University of Kansas, USA (http://www.specifysoftware.org), who developed the ?Specify?-Software. Many thanks also to the curators of different Herbaria: Dr. Regine Jahn (BGBM, FU-Berlin), Dr. Dagmar Triebel (Botanische Staatssammlung München), Dr. Monika Steinhof (Übersee Museum Bremen), Christian Blümel (Universität Rostock) and Dr. Christian Printzen (Forschungsinstitut Senckenberg Frankfurt a.M.). Last not least thanks to the busy co-workers Sebastian Frerichmann, Maas Mollenhauer, Constanze Kovaci and Petra Kretschmann.</abcd:Text> </abcd:Acknowledgement> </abcd:Acknowledgements> <abcd:Citations> <abcd:Citation language="EN"> <abcd:Text>Engels, Monika 2003 - (continuously updated) Catalogus novus et amplificatus speciminum et viventium algarum Desmidiacearum (New and extended catalogue of herbarium specimen and living material of Desmidiaceae in Germany).</abcd:Text> </abcd:Citation> </abcd:Citations> </abcd:IPRStatements> </abcd:Metadata> <abcd:Units> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-6239</abcd:UnitID> <abcd:UnitReferences> <abcd:UnitReference> <abcd:TitleCitation>Algae aquae dulcis exsiccatae praecique scandinavicae, quas adjectis algis marinis chlorophyllaceis et phycochromaceis distribuerunt., Stockholm 1877-1889, Vol. 1-26</abcd:TitleCitation> </abcd:UnitReference> </abcd:UnitReferences> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Mesotaeniaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Zygnematales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Ancylonema nordenskiöldii</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:Identifiers> <abcd:Identifier> <abcd:PersonName> <abcd:FullName>Nordstedt, O.</abcd:FullName> </abcd:PersonName> </abcd:Identifier> </abcd:Identifiers> <abcd:Date> <abcd:DateText>1878-07-18</abcd:DateText> </abcd:Date> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>PreservedSpecimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:MultiMediaObjects> <abcd:MultiMediaObject> <abcd:FileURI>http://ww2.bgbm.org/herbarium/Externe_Bildsammlungen/AlgenEngels/L-Scans-B-Konv/6239.jpg</abcd:FileURI> </abcd:MultiMediaObject> </abcd:MultiMediaObjects> <abcd:Gathering> <abcd:DateTime><abcd:DateText>1878</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Nordstedt, O.</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Brixdal</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>NO</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Nordfjord</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:AreaDetail language="DE">see label</abcd:AreaDetail> <abcd:Biotope> <abcd:Name language="EN">other</abcd:Name> <abcd:Text language="DE">see label</abcd:Text> </abcd:Biotope> <abcd:Notes language="EN">Europe</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: B 4000 08107; Specimen Location: B.</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-3397</abcd:UnitID> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Actinotaenium cucurbita (Brébisson) Teiling ex Ruzicka et Pouzar 1978</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 108, tab. 17: 7a-e as Cosmarium cucurbita; taxon citation: Teiling, E. 1954: Actinotaenium genus Desmidiacearum resuscitatum. - Bot. Notiser 100, p. 406, fig. 66 (invalidy pblished). Validation: Ruzicka, J. &amp; Pouzar, Z. 1978:Erwägungen über die Taxonomie und Nomenklatur der Gattung Actinotaenium TEIL. - Folia Geobot. Phytotax., Praha, 13: p.44</abcd:Notes> </abcd:Identification> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Cosmarium cucurbita (Bréb.) Ralfs 1848</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>false</abcd:PreferredFlag> <abcd:Identifiers> <abcd:Identifier> <abcd:PersonName> <abcd:FullName>Siegmund, W.</abcd:FullName> </abcd:PersonName> </abcd:Identifier> </abcd:Identifiers> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Date> <abcd:DateText>1872-07</abcd:DateText> </abcd:Date> <abcd:Notes>p. 108, tab. 17: 7a-e</abcd:Notes> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>PreservedSpecimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:MultiMediaObjects> <abcd:MultiMediaObject> <abcd:FileURI>http://ww2.bgbm.org/herbarium/Externe_Bildsammlungen/AlgenEngels/L-Scans-HBG-Konv/3397.jpg</abcd:FileURI> </abcd:MultiMediaObject> </abcd:MultiMediaObjects> <abcd:Gathering> <abcd:DateTime><abcd:DateText>1872</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Siegmund, W.</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Reichenberg</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>DE</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Sachsen</abcd:AreaName> </abcd:NamedArea> <abcd:NamedArea> <abcd:AreaName>Dresden</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:AreaDetail language="DE">bei Reichenberg, see label</abcd:AreaDetail> <abcd:Biotope> <abcd:Name language="EN">other</abcd:Name> <abcd:Text language="DE">see label</abcd:Text> </abcd:Biotope> <abcd:Notes language="EN">Europe</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: 8186; Specimen Location: HBG.</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-2483</abcd:UnitID> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Cosmarium turgidum Bréb. 1848</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>false</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 110, tab. 32: 8</abcd:Notes> </abcd:Identification> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Actinotaenium turgidum (Brébisson) Teiling 1954 ex Ruzicka et Pouzar 1978</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>as Cosmarium turgidum p. 110, tab. 32: 8; taxon citation: invalidly published as A. turgidum (Bréb.) Teiling, E. 1954: Actinotaenium genus Desmidiacearum resuscitatum. - Bot. Notiser 100 - p.395, fig.21; Validation by R?ži?ka &amp; Pouzar 1978:Erwägungen über die Taxonomie und Nomenklatur der Gattung Actinotaenium TEIL. - Folia Geobot. Phytotax., Praha, 13: 13 - 66 p.60</abcd:Notes> </abcd:Identification> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </syntaxhighlight> 991cb2bf189d6057e3abe1e89cfa07eeb8644013 924 923 2018-04-10T15:17:30Z DavidFichtmueller 4 wikitext text/x-wiki This page shows a sample response ABCD2 document for a web service with 77 concepts mapped. For easier viewing, you can save the contents of the box into a file with the extension .xml and open that in Firefix or MS Internet Explorer (don't use Safari). You will then be able to collapse the different XML sub trees. <syntaxhighlight lang="xml"> <?xml version='1.0' encoding='UTF-8'?> <biocase:response xmlns:abcd="http://www.tdwg.org/schemas/abcd/2.06" xmlns:biocase="http://www.biocase.org/schemas/protocol/1.3" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.tdwg.org/schemas/abcd/2.06 http://www.bgbm.org/TDWG/CODATA/Schema/ABCD_2.06/ABCD_2.06.XSD http://www.biocase.org/schemas/protocol/1.3 http://www.bgbm.org/biodivinf/schema/protocol_1_31.xsd"> <!-- XML generated by BioCASE PyWrapper software version 2.6.0. Made in Berlin. --> <biocase:header> <biocase:version software="os">nt</biocase:version> <biocase:version software="python">2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]</biocase:version> <biocase:version software="pywrapper">2.6.0</biocase:version> <biocase:version software="dbmod">MS SQL Server module v1.01 using ceODBC ?</biocase:version> <biocase:sendTime>2011-06-29T16:00:47.253000</biocase:sendTime> <biocase:source>AlgenEngels@localhost</biocase:source> <biocase:destination>127.0.0.1</biocase:destination> <biocase:type>search</biocase:type> </biocase:header> <biocase:content recordCount="10" recordDropped="0" recordStart="0" totalSearchHits="11"> <abcd:DataSets> <abcd:DataSet> <abcd:TechnicalContacts> <abcd:TechnicalContact> <abcd:Name>Botanic Garden and Botanical Museum Berlin-Dahlem</abcd:Name> <abcd:Email>...@...</abcd:Email> <abcd:Address>Königin-Luise-Str. 6-8, D-14195 Berlin</abcd:Address> </abcd:TechnicalContact> </abcd:TechnicalContacts> <abcd:ContentContacts> <abcd:ContentContact> <abcd:Name>Dr. ...</abcd:Name> <abcd:Email>...@...</abcd:Email> <abcd:Phone>+49 ...</abcd:Phone> <abcd:Address>..., D-22609 Hamburg</abcd:Address> </abcd:ContentContact> </abcd:ContentContacts> <abcd:Metadata> <abcd:Description> <abcd:Representation language="EN"> <abcd:Title>Desmidiaceae Engels</abcd:Title> <abcd:Details>A new record of reference material of the Desmidiaceae of Germany was compiled, using the databank “Specify”. Aim of the project was to summarize and digitalize the data of the dried and otherwise preserved specimens stored in the German herbaria as well as of the living strains, cultured in culture collections. Many of the references are completed by figures of the labels or by scanned micrographs from preparations for the light- or scanning microscopes.</abcd:Details> <abcd:URI>http://www.gbif.de/botanik/projekte/desmidiaceae_hamburg_ffm/index_html</abcd:URI> </abcd:Representation> </abcd:Description> <abcd:IconURI>http://ww3.bgbm.org/providerResources/BzF.jpg</abcd:IconURI> <abcd:RevisionData> <abcd:Creators>Desmidiaceae Engels (BzF)</abcd:Creators> <abcd:DateModified>2008-01-23T00:00:00</abcd:DateModified> </abcd:RevisionData> <abcd:Owners> <abcd:Owner> <abcd:Organisation> <abcd:Name> <abcd:Representation language="EN"> <abcd:Text>Biozentrum Klein-Flottbek, Universität Hamburg</abcd:Text> <abcd:Abbreviation>BzF</abcd:Abbreviation> </abcd:Representation> </abcd:Name> </abcd:Organisation> </abcd:Owner> </abcd:Owners> <abcd:IPRStatements> <abcd:IPRDeclarations> <abcd:IPRDeclaration language="EN"> <abcd:Text>The Intellectual Property Rights are held by the legal owner or, in case of living persons, by the collector or determinator.</abcd:Text> </abcd:IPRDeclaration> </abcd:IPRDeclarations> <abcd:Copyrights> <abcd:Copyright language="EN"> <abcd:Text>No part of this data base may be copied or reproduced without written permission from the legal owner.</abcd:Text> </abcd:Copyright> </abcd:Copyrights> <abcd:TermsOfUseStatements> <abcd:TermsOfUse language="EN"> <abcd:Text>The use of the data is allowed only for non-profit scientific use and for non-profit nature conservation purpose. The database or part of it may only be used or copied by the written permission from the legal owner. If used for publication, we ask for a copy or an off-print.</abcd:Text> </abcd:TermsOfUse> </abcd:TermsOfUseStatements> <abcd:Disclaimers> <abcd:Disclaimer language="EN"> <abcd:Text>No responsibility is accepted for the accuracy of the information in this database.</abcd:Text> </abcd:Disclaimer> </abcd:Disclaimers> <abcd:Acknowledgements> <abcd:Acknowledgement language="EN"> <abcd:Text>The authors are much indebted to the Scientists of the Informatics Biodiversity Research Center der University of Kansas, USA (http://www.specifysoftware.org), who developed the ?Specify?-Software. Many thanks also to the curators of different Herbaria: Dr. Regine Jahn (BGBM, FU-Berlin), Dr. Dagmar Triebel (Botanische Staatssammlung München), Dr. Monika Steinhof (Übersee Museum Bremen), Christian Blümel (Universität Rostock) and Dr. Christian Printzen (Forschungsinstitut Senckenberg Frankfurt a.M.). Last not least thanks to the busy co-workers Sebastian Frerichmann, Maas Mollenhauer, Constanze Kovaci and Petra Kretschmann.</abcd:Text> </abcd:Acknowledgement> </abcd:Acknowledgements> <abcd:Citations> <abcd:Citation language="EN"> <abcd:Text>Engels, Monika 2003 - (continuously updated) Catalogus novus et amplificatus speciminum et viventium algarum Desmidiacearum (New and extended catalogue of herbarium specimen and living material of Desmidiaceae in Germany).</abcd:Text> </abcd:Citation> </abcd:Citations> </abcd:IPRStatements> </abcd:Metadata> <abcd:Units> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-6239</abcd:UnitID> <abcd:UnitReferences> <abcd:UnitReference> <abcd:TitleCitation>Algae aquae dulcis exsiccatae praecique scandinavicae, quas adjectis algis marinis chlorophyllaceis et phycochromaceis distribuerunt., Stockholm 1877-1889, Vol. 1-26</abcd:TitleCitation> </abcd:UnitReference> </abcd:UnitReferences> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Mesotaeniaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Zygnematales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Ancylonema nordenskiöldii</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:Identifiers> <abcd:Identifier> <abcd:PersonName> <abcd:FullName>Nordstedt, O.</abcd:FullName> </abcd:PersonName> </abcd:Identifier> </abcd:Identifiers> <abcd:Date> <abcd:DateText>1878-07-18</abcd:DateText> </abcd:Date> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>PreservedSpecimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:MultiMediaObjects> <abcd:MultiMediaObject> <abcd:FileURI>http://ww2.bgbm.org/herbarium/Externe_Bildsammlungen/AlgenEngels/L-Scans-B-Konv/6239.jpg</abcd:FileURI> </abcd:MultiMediaObject> </abcd:MultiMediaObjects> <abcd:Gathering> <abcd:DateTime><abcd:DateText>1878</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Nordstedt, O.</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Brixdal</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>NO</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Nordfjord</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:AreaDetail language="DE">see label</abcd:AreaDetail> <abcd:Biotope> <abcd:Name language="EN">other</abcd:Name> <abcd:Text language="DE">see label</abcd:Text> </abcd:Biotope> <abcd:Notes language="EN">Europe</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: B 4000 08107; Specimen Location: B.</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-3397</abcd:UnitID> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Actinotaenium cucurbita (Brébisson) Teiling ex Ruzicka et Pouzar 1978</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 108, tab. 17: 7a-e as Cosmarium cucurbita; taxon citation: Teiling, E. 1954: Actinotaenium genus Desmidiacearum resuscitatum. - Bot. Notiser 100, p. 406, fig. 66 (invalidy pblished). Validation: Ruzicka, J. &amp; Pouzar, Z. 1978:Erwägungen über die Taxonomie und Nomenklatur der Gattung Actinotaenium TEIL. - Folia Geobot. Phytotax., Praha, 13: p.44</abcd:Notes> </abcd:Identification> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Cosmarium cucurbita (Bréb.) Ralfs 1848</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>false</abcd:PreferredFlag> <abcd:Identifiers> <abcd:Identifier> <abcd:PersonName> <abcd:FullName>Siegmund, W.</abcd:FullName> </abcd:PersonName> </abcd:Identifier> </abcd:Identifiers> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Date> <abcd:DateText>1872-07</abcd:DateText> </abcd:Date> <abcd:Notes>p. 108, tab. 17: 7a-e</abcd:Notes> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>PreservedSpecimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:MultiMediaObjects> <abcd:MultiMediaObject> <abcd:FileURI>http://ww2.bgbm.org/herbarium/Externe_Bildsammlungen/AlgenEngels/L-Scans-HBG-Konv/3397.jpg</abcd:FileURI> </abcd:MultiMediaObject> </abcd:MultiMediaObjects> <abcd:Gathering> <abcd:DateTime><abcd:DateText>1872</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Siegmund, W.</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Reichenberg</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>DE</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Sachsen</abcd:AreaName> </abcd:NamedArea> <abcd:NamedArea> <abcd:AreaName>Dresden</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:AreaDetail language="DE">bei Reichenberg, see label</abcd:AreaDetail> <abcd:Biotope> <abcd:Name language="EN">other</abcd:Name> <abcd:Text language="DE">see label</abcd:Text> </abcd:Biotope> <abcd:Notes language="EN">Europe</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: 8186; Specimen Location: HBG.</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-2483</abcd:UnitID> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Cosmarium turgidum Bréb. 1848</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>false</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 110, tab. 32: 8</abcd:Notes> </abcd:Identification> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Actinotaenium turgidum (Brébisson) Teiling 1954 ex Ruzicka et Pouzar 1978</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>as Cosmarium turgidum p. 110, tab. 32: 8; taxon citation: invalidly published as A. turgidum (Bréb.) Teiling, E. 1954: Actinotaenium genus Desmidiacearum resuscitatum. - Bot. Notiser 100 - p.395, fig.21; Validation by R?ži?ka &amp; Pouzar 1978:Erwägungen über die Taxonomie und Nomenklatur der Gattung Actinotaenium TEIL. - Folia Geobot. Phytotax., Praha, 13: 13 - 66 p.60</abcd:Notes> </abcd:Identification> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Pleurotaenium turgidum De Bary 1858</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>false</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>De Bary, A: Untersuchungen über die Familie der Conjugaten (Zygnemeen und Desmidieen)., Leipzig 1858</abcd:TitleCitation> <abcd:CitationDetail>p. 91</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 75, tab. 5: 31; invalidly published</abcd:Notes> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>PreservedSpecimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:Gathering> <abcd:DateTime><abcd:DateText>0</abcd:DateText> </abcd:DateTime> <abcd:LocalityText language="DE">Neukirch</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>DE</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Sachsen</abcd:AreaName> </abcd:NamedArea> <abcd:NamedArea> <abcd:AreaName>Chemnitz</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:AreaDetail language="DE">Oberlausitz, Neukirch</abcd:AreaDetail> <abcd:Notes language="EN">Europe</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: 7474; Specimen Location: HBG. together with Closterium hybridum, Pl. truncatum</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-4603</abcd:UnitID> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Arthrodesmus octocornis var. major Roy &amp; Biss. 1894</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:Identifiers> <abcd:Identifier> <abcd:PersonName> <abcd:FullName>Richter, Paul</abcd:FullName> </abcd:PersonName> </abcd:Identifier> </abcd:Identifiers> <abcd:Notes>see label</abcd:Notes> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>PreservedSpecimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:MultiMediaObjects> <abcd:MultiMediaObject> <abcd:FileURI>http://ww2.bgbm.org/herbarium/Externe_Bildsammlungen/AlgenEngels/L-Scans-HBG-Konv/4602.jpg</abcd:FileURI> </abcd:MultiMediaObject> </abcd:MultiMediaObjects> <abcd:Gathering> <abcd:DateTime><abcd:DateText>0</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Richter, Paul</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Leipzig</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>DE</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Sachsen</abcd:AreaName> </abcd:NamedArea> <abcd:NamedArea> <abcd:AreaName>Leipzig</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:Notes language="EN">Europe</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: 8773; Specimen Location: HBG.</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-6764</abcd:UnitID> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Actinotaenium curtum var. attenuatum (Bréb.) Teil. ex Ruzicka &amp; Pouzar 1978</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>as Cosmarium attenuatum p. 110, tab. 17: 9; taxon citation: Teiling, E., 1954: Actinotaenium genus Desmidiacearum resuscitatum. - Bot. Not., Lund, 1954, p. 391, fig. 2; invaldly published; Validation: Ruzicka, J., &amp; Pouzar, Z., 1978: Erwägungen über die Taxonomie und Nomenklatur der Gattung ActinotaeniumTEIL. - Folia geobot. Phytotax., Praha, 13: 33-66., p.47</abcd:Notes> </abcd:Identification> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Cosmarium attenuatum Ralfs 1848</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>false</abcd:PreferredFlag> <abcd:Identifiers> <abcd:Identifier> <abcd:PersonName> <abcd:FullName>Richter, Paul</abcd:FullName> </abcd:PersonName> </abcd:Identifier> </abcd:Identifiers> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 110, tab. 17: 9a-d</abcd:Notes> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>PreservedSpecimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:Gathering> <abcd:DateTime><abcd:DateText>1898</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Rogers,</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Ennepetal-Milspe</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>DE</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Nordrhein-Westfalen</abcd:AreaName> </abcd:NamedArea> <abcd:NamedArea> <abcd:AreaName>Hagen</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:Biotope> <abcd:Text language="DE">an einer Felsenwand</abcd:Text> </abcd:Biotope> <abcd:Notes language="EN">Europe</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: B 4000 08307; Specimen Location: B.</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-3698</abcd:UnitID> <abcd:UnitReferences> <abcd:UnitReference> <abcd:TitleCitation>Algae aquae dulcis exsiccatae praecique scandinavicae, quas adjectis algis marinis chlorophyllaceis et phycochromaceis distribuerunt., Lund 1896-1903, Vol. 27-34</abcd:TitleCitation> </abcd:UnitReference> </abcd:UnitReferences> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Actinotaenium palangula (Brébisson) Teiling 1954 ex Ruzicka et Pouzar 1978</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 212; original figure: Brébisson 1856: p.132, tab.1: 21; taxon citation: Ruzicka, J. &amp; Pouzar, Z., 1978: Erwägungen über die Taxonomie und Nomenklatur der Gattung Actinotaenium TEIL. - Folia Geobot. Phytotax., Praha, 13: p.46</abcd:Notes> </abcd:Identification> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Cosmarium palangula Brébisson in Ralfs 1848</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>false</abcd:PreferredFlag> <abcd:Identifiers> <abcd:Identifier> <abcd:PersonName> <abcd:FullName>Bohlin, Knut</abcd:FullName> </abcd:PersonName> </abcd:Identifier> </abcd:Identifiers> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Date> <abcd:DateText>1898-07-06</abcd:DateText> </abcd:Date> <abcd:Notes>p. 212; original figure: Brébisson 1856: p.132, tab.1: 21</abcd:Notes> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>PreservedSpecimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:MultiMediaObjects> <abcd:MultiMediaObject> <abcd:FileURI>http://ww2.bgbm.org/herbarium/Externe_Bildsammlungen/AlgenEngels/L-Scans-HBG-Konv/3690-3703.jpg</abcd:FileURI> </abcd:MultiMediaObject> </abcd:MultiMediaObjects> <abcd:Gathering> <abcd:DateTime><abcd:DateText>1898</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Bohlin, Knut</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Pico da Cruz</abcd:LocalityText> <abcd:Notes language="EN">Africa</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: 8328; Specimen Location: HBG. sample contains: Cosmarium ralfsii var. azoricum, Cosmarium palangula, Cosmarium pyramidatum, Cosmarium venustum, Euastru didelta f. scorbiculata, Staurastrum muticum var. depressum, Staurastrum scabrum</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-4569</abcd:UnitID> <abcd:UnitReferences> <abcd:UnitReference> <abcd:TitleCitation>Die Algen Europa's (Fortsetzung der Algen Sachsens, resp. Mittel-Europa's)., Dresden 1861 - 1879, Vol. Dec. 101 - 259</abcd:TitleCitation> </abcd:UnitReference> </abcd:UnitReferences> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Staurodesmus convergens (Ehrenberg ex Ralfs) Teiling 1967</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 118, tab. 20: 3a-d; taxon citation: Teiling, E., 1948: Staurodesmus, genus novum.Containing monospinous desmids. - Bot. Not., Lund, 1948: p. 57</abcd:Notes> </abcd:Identification> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Arthrodesmus convergens Ehrenberg ex Ralfs 1848</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>false</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 118, tab. 20: 3a-d</abcd:Notes> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>PreservedSpecimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:Gathering> <abcd:DateTime><abcd:DateText>0</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Rostock, M.</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Dretschen</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>DE</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Sachsen</abcd:AreaName> </abcd:NamedArea> <abcd:NamedArea> <abcd:AreaName>Bautzen</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:Notes language="EN">Europe</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: 8751; Specimen Location: HBG.</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-7071</abcd:UnitID> <abcd:UnitReferences> <abcd:UnitReference> <abcd:TitleCitation>Kryptogamae exsiccatae editae a museo hist. natur. vindobonensi., Wien 1894 ff</abcd:TitleCitation> </abcd:UnitReference> </abcd:UnitReferences> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Arthrodesmus quadricaudatus</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:Identifiers> <abcd:Identifier> <abcd:PersonName> <abcd:FullName>Hansgirg, A.</abcd:FullName> </abcd:PersonName> </abcd:Identifier> </abcd:Identifiers> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>PreservedSpecimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:Gathering> <abcd:DateTime><abcd:DateText>0</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Hansgirg, A.</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Bombay</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>IN</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Bombay</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:Biotope> <abcd:Name language="EN">other</abcd:Name> <abcd:Text language="DE">see label</abcd:Text> </abcd:Biotope> <abcd:Notes language="EN">Asia</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: B 4000 08385; Specimen Location: B. sample contains: Cosmarium nitidulum, Arthrodesmus quadricaudatus</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-6250</abcd:UnitID> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Staurodesmus incus (Brébisson ex Ralfs) Teiling 1967</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 118, tab. 20: 4 as Arthrodesmus incus; taxon citation: Teiling, E., 1967: The desmid genus Staurodesmus. A taxonomic study. Ark. f. Bot. II, 6(11), p. p. 511, tab. 5: 9</abcd:Notes> </abcd:Identification> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Arthrodesmus incus (Bréb.) Hassall ex Ralfs 1848</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>false</abcd:PreferredFlag> <abcd:Identifiers> <abcd:Identifier> <abcd:PersonName> <abcd:FullName>Borge, O.</abcd:FullName> </abcd:PersonName> </abcd:Identifier> </abcd:Identifiers> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Date> <abcd:DateText>1889-08-08</abcd:DateText> </abcd:Date> <abcd:Notes>p. 118, tab. 20: 4a-l</abcd:Notes> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>PreservedSpecimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:Gathering> <abcd:DateTime><abcd:DateText>1889</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Borge, O.</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Väddö</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>SE</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Upland</abcd:AreaName> </abcd:NamedArea> <abcd:NamedArea> <abcd:AreaName>Väddö</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:Notes language="EN">Europe</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: B 4000 32088; Specimen Location: B.</abcd:Notes> </abcd:Unit> <abcd:Unit> <abcd:SourceInstitutionID>BzF</abcd:SourceInstitutionID> <abcd:SourceID>Desmidiaceae Engels</abcd:SourceID> <abcd:UnitID>Main-1-6897</abcd:UnitID> <abcd:Identifications> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Actinotaenium cucurbita (Brébisson) Teiling ex Ruzicka et Pouzar 1978</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>true</abcd:PreferredFlag> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Notes>p. 108, tab. 17: 7a-e as Cosmarium cucurbita; taxon citation: Teiling, E. 1954: Actinotaenium genus Desmidiacearum resuscitatum. - Bot. Notiser 100, p. 406, fig. 66 (invalidy pblished). Validation: Ruzicka, J. &amp; Pouzar, Z. 1978:Erwägungen über die Taxonomie und Nomenklatur der Gattung Actinotaenium TEIL. - Folia Geobot. Phytotax., Praha, 13: p.44</abcd:Notes> </abcd:Identification> <abcd:Identification> <abcd:Result> <abcd:TaxonIdentified> <abcd:HigherTaxa> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiaceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>familia</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Desmidiales</abcd:HigherTaxonName> <abcd:HigherTaxonRank>ordo</abcd:HigherTaxonRank> </abcd:HigherTaxon> <abcd:HigherTaxon> <abcd:HigherTaxonName>Conjugatophyceae</abcd:HigherTaxonName> <abcd:HigherTaxonRank>classis</abcd:HigherTaxonRank> </abcd:HigherTaxon> </abcd:HigherTaxa> <abcd:ScientificName> <abcd:FullScientificNameString>Cosmarium cucurbita (Bréb.) Ralfs 1848</abcd:FullScientificNameString> </abcd:ScientificName> </abcd:TaxonIdentified> </abcd:Result> <abcd:PreferredFlag>false</abcd:PreferredFlag> <abcd:Identifiers> <abcd:Identifier> <abcd:PersonName> <abcd:FullName>Bulnheim, O.</abcd:FullName> </abcd:PersonName> </abcd:Identifier> </abcd:Identifiers> <abcd:References> <abcd:Reference> <abcd:TitleCitation>Ralfs, J: The British Desmidieae, London 1848</abcd:TitleCitation> <abcd:CitationDetail>p. 22 + 226, 35 pl.</abcd:CitationDetail> </abcd:Reference> </abcd:References> <abcd:Date> <abcd:DateText>1883-07</abcd:DateText> </abcd:Date> <abcd:Notes>p. 108, tab. 17: 7a-e</abcd:Notes> </abcd:Identification> </abcd:Identifications> <abcd:RecordBasis>PreservedSpecimen</abcd:RecordBasis> <abcd:SpecimenUnit> <abcd:Preparations> <abcd:Preparation> <abcd:PreparationType>preserved material</abcd:PreparationType> </abcd:Preparation> </abcd:Preparations> </abcd:SpecimenUnit> <abcd:MultiMediaObjects> <abcd:MultiMediaObject> <abcd:FileURI>http://ww2.bgbm.org/herbarium/Externe_Bildsammlungen/AlgenEngels/L-Scans-B-Konv/6897-6899.jpg</abcd:FileURI> </abcd:MultiMediaObject> </abcd:MultiMediaObjects> <abcd:Gathering> <abcd:DateTime><abcd:DateText>1883</abcd:DateText> </abcd:DateTime> <abcd:Agents> <abcd:GatheringAgent> <abcd:Person> <abcd:FullName>Bulnheim, O.</abcd:FullName> </abcd:Person> </abcd:GatheringAgent> </abcd:Agents> <abcd:LocalityText language="DE">Leipzig</abcd:LocalityText> <abcd:Country> <abcd:ISO3166Code>DE</abcd:ISO3166Code> </abcd:Country> <abcd:NamedAreas> <abcd:NamedArea> <abcd:AreaName>Sachsen</abcd:AreaName> </abcd:NamedArea> <abcd:NamedArea> <abcd:AreaName>Leipzig</abcd:AreaName> </abcd:NamedArea> </abcd:NamedAreas> <abcd:AreaDetail language="DE">see label</abcd:AreaDetail> <abcd:Notes language="EN">Europe</abcd:Notes> </abcd:Gathering> <abcd:Notes>Barcode: B 4000 08358; Specimen Location: B. sample contains: Cosmarium cucurbita, Euastrum didelta, Closterium intermedium</abcd:Notes> </abcd:Unit> </abcd:Units> </abcd:DataSet> </abcd:DataSets> </biocase:content> <biocase:diagnostics> <biocase:diagnostic severity="INFO">Datasource wrapper AlgenEngels requested</biocase:diagnostic> <biocase:diagnostic severity="INFO">Reading PSF from C:\Workspace\bps2\config\datasources\AlgenEngels\provider_setup_file.xml</biocase:diagnostic> <biocase:diagnostic severity="DEBUG">PSF: PSF=C:\Workspace\bps2\config\datasources\AlgenEngels\provider_setup_file.xml, recLimit=0, loglevel=40, user=WebUser, database=AlgenEngels, dbIP=, dbms=odbc_tsql, encoding=latin_1, schemas={u'http://www.tdwg.org/schemas/abcd/2.06': &lt;biocase.wrapper.psf_handler.SupportedSchema instance at 0x00EB1E90&gt;}, tablegraph=GRAPH: graph: meta, determination-taxonNames, higherTaxon-taxonNames, images-specimen, namedAreas-collectingEvent, siteFeatures-collectingEvent, specimen-determination, specimen-collectingEvent, unitReference-specimen, +++ ALIAS2TABLE: {u'collectingEvent': u'abcd_CollectingEvent', u'taxonNames': u'abcd_TaxonName', u'higherTaxon': u'abcd_HigherTaxa', u'specimen': u'abcd_Object', u'siteFeatures': u'abcd_SiteFeatureTerms', u'meta': u'metadata', u'determination': u'abcd_Determination', u'images': u'abcd_Images', u'unitReference': u'abcd_UnitReference', u'namedAreas': u'abcd_NamedAreaNames'}</biocase:diagnostic> </biocase:diagnostics> </biocase:response> </syntaxhighlight> 9e0f1446edbff6afe62954f75c3493f4798b7902 0 6 925 806 2018-04-12T09:11:47Z JoergHoletschek 3 /* Installing Apache/IIS and Python */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements: # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 2.5+ to run, it doesn't work with Python 3. On most Linux systems, Python 2.7 will be pre-installed, and this works fine with BioCASe. By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html === Testing the Link === Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages/Java == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! === Updating from version 2.5 or later === # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. === Updating from older versions < 2.5 === # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. fb3d51db23f46968b6034a61a57259fab0fe9b2e 932 925 2018-05-25T13:13:48Z JoergHoletschek 3 wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation (see section below), you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you're famliar, we'd suggest to use the Docker path for installing BioCASe (read more in section three below). If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 2.5+ to run, it doesn't work with Python 3. On most Linux systems, Python 2.7 will be pre-installed, and this works fine with BioCASe. By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html === Testing the Link === Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages/Java == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! === Updating from version 2.5 or later === # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. === Updating from older versions < 2.5 === # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. 708cebe0de4c3fd70d60607b56e4a4347c0c6001 933 932 2018-05-25T13:22:35Z JoergHoletschek 3 wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation (see section below), you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you're famliar, we'd suggest to use the Docker path for installing BioCASe (read more in section three below). If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section three). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, since you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). == Installing Apache/IIS and Python == If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 2.5+ to run, it doesn't work with Python 3. On most Linux systems, Python 2.7 will be pre-installed, and this works fine with BioCASe. By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. == Downloading the Provider Software == You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. === Downloading the Archive File === Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. === Downloading from the Subversion Repository === If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] == Running the Setup Script == The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] == Linking BioCASe and Your Web Server == The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. === Apache === If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. === Internet Information Server === If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html === Testing the Link === Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] == Installing Additional Python Packages/Java == Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. == Changing the Default Password == The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. == Checking Your Installation == Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! === Updating from version 2.5 or later === # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. === Updating from older versions < 2.5 === # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. 3973af83b0de5f339d5d479f839d6b116d68a8dd 934 933 2018-05-25T13:25:36Z JoergHoletschek 3 wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation (see section below), you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you're famliar, we'd suggest to use the Docker path for installing BioCASe (read more in section three below). If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section three). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, since you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 2.5+ to run, it doesn't work with Python 3. On most Linux systems, Python 2.7 will be pre-installed, and this works fine with BioCASe. By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Installing BioCASe from a Docker image == == Updating an existing BioCASe Installation == '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! === Updating from version 2.5 or later === # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. === Updating from older versions < 2.5 === # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. cbbcbb2639eedc7b7d3d052eed249a33c4d87fd6 935 934 2018-05-25T13:28:25Z JoergHoletschek 3 /* Requirements and Considerations */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation (see section below), you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you're famliar, we'd suggest to use the Docker path for installing BioCASe (read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below). If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section three). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, since you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 2.5+ to run, it doesn't work with Python 3. On most Linux systems, Python 2.7 will be pre-installed, and this works fine with BioCASe. By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Installing BioCASe from a Docker image == == Updating an existing BioCASe Installation == '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! === Updating from version 2.5 or later === # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. === Updating from older versions < 2.5 === # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. d8f0e82f8dfafae75bea2d043743e20608ccda47 936 935 2018-05-25T13:30:01Z JoergHoletschek 3 wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation (see section below), you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you're famliar, we'd suggest to use the Docker path for installing BioCASe (read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below). If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, since you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 2.5+ to run, it doesn't work with Python 3. On most Linux systems, Python 2.7 will be pre-installed, and this works fine with BioCASe. By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Installing BioCASe from a Docker image == == Updating an existing BioCASe Installation == '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! === Updating from version 2.5 or later === # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. === Updating from older versions < 2.5 === # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. 815387baa31f757258487dc9d88470880fe3c12a 937 936 2018-05-25T13:32:19Z JoergHoletschek 3 wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation (see section below), you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you're famliar, we'd suggest to use the Docker path for installing BioCASe (read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below). If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, since you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 2.5+ to run, it doesn't work with Python 3. On most Linux systems, Python 2.7 will be pre-installed, and this works fine with BioCASe. By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Installing BioCASe from a Docker image == == Updating an existing BioCASe Installation == === Updating a regular BioCASe installtion === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. === Updating a BioCASe Docker installation === 8abb7e4529d27ea62466bd738ac4c537cfa3b50e 938 937 2018-05-25T13:34:55Z JoergHoletschek 3 /* Requirements and Considerations */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation (see section below), you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, since you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 2.5+ to run, it doesn't work with Python 3. On most Linux systems, Python 2.7 will be pre-installed, and this works fine with BioCASe. By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Installing BioCASe from a Docker image == == Updating an existing BioCASe Installation == === Updating a regular BioCASe installtion === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. === Updating a BioCASe Docker installation === e9128c7b38aab4cc89ddace5c6f972c378914183 939 938 2018-05-25T13:35:22Z JoergHoletschek 3 /* Updating a regular BioCASe installtion */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation (see section below), you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, since you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 2.5+ to run, it doesn't work with Python 3. On most Linux systems, Python 2.7 will be pre-installed, and this works fine with BioCASe. By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Installing BioCASe from a Docker image == == Updating an existing BioCASe Installation == === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. === Updating a BioCASe Docker installation === b842ff8770bffb3c150626fd17eb7fbd1af7c852 940 939 2018-05-25T13:44:32Z JoergHoletschek 3 /* Installing BioCASe from a Docker image */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation (see section below), you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, since you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 2.5+ to run, it doesn't work with Python 3. On most Linux systems, Python 2.7 will be pre-installed, and this works fine with BioCASe. By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the installed components step-by-step. In addition, it's easier to update an exisiting installation. == Updating an existing BioCASe Installation == === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. === Updating a BioCASe Docker installation === e4b18efc2b1e0412e5b27cd044268f8cfb5a9f72 941 940 2018-05-25T13:57:34Z JoergHoletschek 3 /* Installing BioCASe from a Docker image */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation (see section below), you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, since you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 2.5+ to run, it doesn't work with Python 3. On most Linux systems, Python 2.7 will be pre-installed, and this works fine with BioCASe. By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the installed components step-by-step. In addition, it's easier to update an exisiting installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. == Updating an existing BioCASe Installation == === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. === Updating a BioCASe Docker installation === 7660e8c07a2f8f3cb61e5d8b39ec2a9ff5bf52e7 942 941 2018-05-25T14:03:16Z JoergHoletschek 3 /* Installing BioCASe from a Docker image */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation (see section below), you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, since you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 2.5+ to run, it doesn't work with Python 3. On most Linux systems, Python 2.7 will be pre-installed, and this works fine with BioCASe. By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the installed components step-by-step. In addition, it's easier to update an exisiting installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the Makefile that contains the commands for installing and updating BioCASe. Save it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install''. == Updating an existing BioCASe Installation == === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. === Updating a BioCASe Docker installation === 640562737a9e9dd0fcfb1de6c3e8a37a425174c7 943 942 2018-05-25T14:10:01Z JoergHoletschek 3 /* Installing BioCASe from a Docker image */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation (see section below), you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, since you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 2.5+ to run, it doesn't work with Python 3. On most Linux systems, Python 2.7 will be pre-installed, and this works fine with BioCASe. By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the installed components step-by-step. In addition, it's easier to update an exisiting installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the Makefile that contains the commands for installing and updating BioCASe. Save it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it: <Bild> Per default, BioCASe will use port 80. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8080, for example, type in ''make install PORT=8080'' instead. Voilà, that's it! To check your BioCASe installation, go to http://localhost/biocase. == Updating an existing BioCASe Installation == === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. === Updating a BioCASe Docker installation === ffe9adfeb5890a9e3658bd80fd20c08cfdf41559 944 943 2018-05-25T14:13:06Z JoergHoletschek 3 /* Installing BioCASe from a Docker image */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation (see section below), you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, since you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 2.5+ to run, it doesn't work with Python 3. On most Linux systems, Python 2.7 will be pre-installed, and this works fine with BioCASe. By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the installed components step-by-step. In addition, it's easier to update an exisiting installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Dockerfile Makefile] that contains the commands for installing and updating BioCASe. Save it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it: <Bild> Per default, BioCASe will use port 80. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8080, for example, type in ''make install PORT=8080'' instead. Voilà, that's it! To check your BioCASe installation, go to http://localhost/biocase. == Updating an existing BioCASe Installation == === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. === Updating a BioCASe Docker installation === a94a5e7c7a5609b88e7a3e4ec48d552915619ced 945 944 2018-05-25T14:14:00Z JoergHoletschek 3 /* Installing BioCASe from a Docker image */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation (see section below), you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, since you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 2.5+ to run, it doesn't work with Python 3. On most Linux systems, Python 2.7 will be pre-installed, and this works fine with BioCASe. By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the installed components step-by-step. In addition, it's easier to update an exisiting installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Dockerfile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it: <Bild> Per default, BioCASe will use port 80. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8080, for example, type in ''make install PORT=8080'' instead. Voilà, that's it! To check your BioCASe installation, go to http://localhost/biocase. == Updating an existing BioCASe Installation == === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. === Updating a BioCASe Docker installation === 480f3a739492980a12c8f3161f7c36843ef550cb 946 945 2018-05-25T14:15:31Z JoergHoletschek 3 /* Installing BioCASe from a Docker image */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation (see section below), you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, since you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 2.5+ to run, it doesn't work with Python 3. On most Linux systems, Python 2.7 will be pre-installed, and this works fine with BioCASe. By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the installed components step-by-step. In addition, it's easier to update an exisiting installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Dockerfile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it: <Bild> Per default, BioCASe will use port 80. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8080, for example, type in ''make install PORT=8080'' instead. Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. == Updating an existing BioCASe Installation == === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. === Updating a BioCASe Docker installation === e8b50b002cd74c52f46cfc0ae90ad703e1fa8c07 947 946 2018-05-25T14:16:53Z JoergHoletschek 3 /* Installing BioCASe from a Docker image */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation (see section below), you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, since you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 2.5+ to run, it doesn't work with Python 3. On most Linux systems, Python 2.7 will be pre-installed, and this works fine with BioCASe. By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the installed components step-by-step. In addition, it's easier to update an exisiting installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Dockerfile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it: <Bild> Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8080, for example, type in ''make install PORT=8080'' instead. Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. == Updating an existing BioCASe Installation == === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. === Updating a BioCASe Docker installation === 111fe4c605be664be5ec306f7d236d7a490641ae 948 947 2018-05-25T14:19:45Z JoergHoletschek 3 /* Installing BioCASe from a Docker image */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation (see section below), you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, since you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 2.5+ to run, it doesn't work with Python 3. On most Linux systems, Python 2.7 will be pre-installed, and this works fine with BioCASe. By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the installed components step-by-step. In addition, it's easier to update an exisiting installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Dockerfile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it.: <Bild> Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8080, for example, type in ''make install PORT=8080'' instead. Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. == Updating an existing BioCASe Installation == === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. === Updating a BioCASe Docker installation === 363168c045f27ac0b70709c471e0b26d85527899 949 948 2018-05-25T14:48:55Z JoergHoletschek 3 /* Installing BioCASe from a Docker image */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation (see section below), you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, since you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 2.5+ to run, it doesn't work with Python 3. On most Linux systems, Python 2.7 will be pre-installed, and this works fine with BioCASe. By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the installed components step-by-step. In addition, it's easier to update an exisiting installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Dockerfile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead. Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. == Updating an existing BioCASe Installation == === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. === Updating a BioCASe Docker installation === 6eafd890625dc8bd6c1c0f2711fce30f7454d196 950 949 2018-05-25T14:49:32Z JoergHoletschek 3 /* Installing BioCASe from a Docker image */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation (see section below), you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, since you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 2.5+ to run, it doesn't work with Python 3. On most Linux systems, Python 2.7 will be pre-installed, and this works fine with BioCASe. By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the installed components step-by-step. In addition, it's easier to update an exisiting installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Dockerfile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. == Updating an existing BioCASe Installation == === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. === Updating a BioCASe Docker installation === d16d5239441924ef96dc98b0bb34de0da129da52 952 950 2018-05-25T14:52:20Z JoergHoletschek 3 /* Installing BioCASe from a Docker image */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation (see section below), you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, since you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 2.5+ to run, it doesn't work with Python 3. On most Linux systems, Python 2.7 will be pre-installed, and this works fine with BioCASe. By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the installed components step-by-step. In addition, it's easier to update an exisiting installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Dockerfile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|600px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. == Updating an existing BioCASe Installation == === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. === Updating a BioCASe Docker installation === decaf05ddcd038f132a0d64ec4cf9137b9328939 953 952 2018-05-25T14:54:41Z JoergHoletschek 3 /* Installing BioCASe from a Docker image */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation (see section below), you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, since you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 2.5+ to run, it doesn't work with Python 3. On most Linux systems, Python 2.7 will be pre-installed, and this works fine with BioCASe. By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the installed components step-by-step. In addition, it's easier to update an exisiting installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Dockerfile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|600px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. You can also use the Makefile for stopping and starting the BioCASe container - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. == Updating an existing BioCASe Installation == === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. === Updating a BioCASe Docker installation === 2d7e3edfd42e9f608654866faa8e4c85297939c5 954 953 2018-05-25T14:55:36Z JoergHoletschek 3 /* Installing BioCASe from a Docker image */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation (see section below), you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, since you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 2.5+ to run, it doesn't work with Python 3. On most Linux systems, Python 2.7 will be pre-installed, and this works fine with BioCASe. By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the installed components step-by-step. In addition, it's easier to update an exisiting installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|600px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. You can also use the Makefile for stopping and starting the BioCASe container - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. == Updating an existing BioCASe Installation == === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. === Updating a BioCASe Docker installation === 2d8d71c50fd9c2ccb1c177a4abc1da170be83cd0 955 954 2018-05-25T15:06:35Z JoergHoletschek 3 /* Installing BioCASe from a Docker image */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation (see section below), you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, since you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 2.5+ to run, it doesn't work with Python 3. On most Linux systems, Python 2.7 will be pre-installed, and this works fine with BioCASe. By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the installed components step-by-step. In addition, it's easier to update an exisiting installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|600px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. You can also use the Makefile for stopping and starting the BioCASe container - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be store in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Updating an existing BioCASe Installation == === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. === Updating a BioCASe Docker installation === 37f69e80d578384a6d9b80350815711ba446873a 956 955 2018-05-25T15:09:53Z JoergHoletschek 3 /* Updating a BioCASe Docker installation */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation (see section below), you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, since you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 2.5+ to run, it doesn't work with Python 3. On most Linux systems, Python 2.7 will be pre-installed, and this works fine with BioCASe. By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the installed components step-by-step. In addition, it's easier to update an exisiting installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|600px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. You can also use the Makefile for stopping and starting the BioCASe container - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be store in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Updating an existing BioCASe Installation == === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. === Updating a BioCASe Docker installation === This is very simple with the Mss you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. efa530c646db60d4a50f922d9786e239e8856e5e 957 956 2018-05-25T15:10:24Z JoergHoletschek 3 /* Updating a BioCASe Docker installation */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation (see section below), you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, since you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 2.5+ to run, it doesn't work with Python 3. On most Linux systems, Python 2.7 will be pre-installed, and this works fine with BioCASe. By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the installed components step-by-step. In addition, it's easier to update an exisiting installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|600px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. You can also use the Makefile for stopping and starting the BioCASe container - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be store in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Updating an existing BioCASe Installation == === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. f27d3906bfc5b555773fcb24b3b70721c90a1436 958 957 2018-05-25T15:13:31Z JoergHoletschek 3 /* Installing BioCASe from a Docker image */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation (see section below), you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, since you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 2.5+ to run, it doesn't work with Python 3. On most Linux systems, Python 2.7 will be pre-installed, and this works fine with BioCASe. By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the installed components step-by-step. In addition, it's easier to update an exisiting installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|600px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[http://wiki.bgbm.org/bps/index.php/Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be store in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Updating an existing BioCASe Installation == === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. 8a074f4edf76a591a89108a69b6a757505ad9d29 959 958 2018-05-25T15:17:36Z JoergHoletschek 3 /* Installing BioCASe from a Docker image */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation (see section below), you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, since you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 2.5+ to run, it doesn't work with Python 3. On most Linux systems, Python 2.7 will be pre-installed, and this works fine with BioCASe. By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the installed components step-by-step. In addition, it's easier to update an exisiting installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|600px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be store in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Updating an existing BioCASe Installation == === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. 5cf8186f7687f00bec46f63002aed115998dda9b 960 959 2018-05-28T08:47:02Z JoergHoletschek 3 /* Installing BioCASe from a Docker image */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation (see section below), you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, since you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 2.5+ to run, it doesn't work with Python 3. On most Linux systems, Python 2.7 will be pre-installed, and this works fine with BioCASe. By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an exisiting installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|600px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be store in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Updating an existing BioCASe Installation == === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. 431712c82ce4f832355b02b2bc391219091ba24c 961 960 2018-05-28T08:50:27Z JoergHoletschek 3 /* Installing BioCASe from a Docker image */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation (see section below), you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, since you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 2.5+ to run, it doesn't work with Python 3. On most Linux systems, Python 2.7 will be pre-installed, and this works fine with BioCASe. By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an exisiting installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|600px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be store in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Updating an existing BioCASe Installation == === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. 43e3f5624f6f1372acffd0b1623a10522dd37e49 964 961 2018-05-28T09:06:22Z JoergHoletschek 3 wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an exisiting installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|600px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be store in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, since you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 2.5+ to run, it doesn't work with Python 3. On most Linux systems, Python 2.7 will be pre-installed, and this works fine with BioCASe. By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. 405355fe64c57f14d10caae114d5c265c97b8e47 965 964 2018-05-28T09:37:31Z JoergHoletschek 3 /* Installing BioCASe from a Docker image */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an exisiting installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|650px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be store in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, since you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 2.5+ to run, it doesn't work with Python 3. On most Linux systems, Python 2.7 will be pre-installed, and this works fine with BioCASe. By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. 249404fc5d0df05f07847d9b043e5ff1558a4f30 966 965 2018-05-28T09:38:27Z JoergHoletschek 3 /* Installing BioCASe from a Docker image */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an exisiting installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|650px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, since you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 2.5+ to run, it doesn't work with Python 3. On most Linux systems, Python 2.7 will be pre-installed, and this works fine with BioCASe. By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. f41fdac2039ca8ab68499ac27941df1168dbdeaa 6 114 951 2018-05-25T14:50:09Z JoergHoletschek 3 wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 0 49 962 871 2018-05-28T08:56:43Z JoergHoletschek 3 /* Requirements for Using the BPS */ wikitext text/x-wiki [[http://wiki.bgbm.org/bps/index.php/BeginnersGuide_de Diese Seite auf Deutsch]] == What is the BioCASe Provider Software? == The BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. The Provider Software is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. The Provider Software is a suite of tools that need to be installed on the data provider’s web server. The core component is the PyWrapper, an XML/CGI database interface written in Python that allows a standardized access to a variety of database management systems and arbitrarily structured databases. Grouped around this are a number of tools for configuring, testing and debugging a BioCASe installation or web service. Even though BioCASe can be used for any conceptual XML data schema, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the [http://www.biocase.org BioCASe network] and the [http://www.gbif.org Global Biodiversity Information Facility]. == Requirements for Using the BPS == Before using the BioCASe Provider Software, four requirements need to be checked: * For one thing, the data to be published needs to be stored in an SQL compliant relational database management system (DBMS), with the data model being well documented or at least knowledge about the data model existing on someone’s head. Non-SQL DBMS are not supported. Currently, the BioCASe Provider Software can connect to the following databases: Microsoft Access, Microsoft SQL Server, MySQL, Postgres, Oracle, Foxpro, Sybase, 4D, DB2, Firebird. * For installing BioCASe, you need to have a web server (Apache or Microsoft IIS) that is permanently connected to the Internet. This can be an institutional server or a hosting service (even though it’s usually a pain to get privileges to install software for the latter). Dial-up connections won’t work. * For the web server in question, you must have the possibility to install Python (if it is not installed yet) and Python packages. Even though this can be done by the IT staff, you should make sure that there are no institutional policies colliding. If you want to create DarwinCore Archives, a Java Runtime Environment is required. * A direct connection can be established from the web server that will run the BPS to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL, 1433 for SQL Server). BioCASe can now be deployed as a Docker container, which makes it very to install, update and remove BioCASe. (Docker is an operating-system-level virtualization software that allows easy software deployment; if you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide].) == Steps == Once you’ve made sure the BioCASe Provider Software is the right choice for you ([[Special:Contact|contact the BioCASe team]] if you feel unsafe with your decision), you need to do the following steps: # Think about which information you want to publish to the network. This involves settling the question of ownership of the data and the terms of use, license and usage restrictions for the published data. Also you should consider which information should '''not''' be published because they might affect endangered species (options are to block certain occurrences from publication or to blur the exact locality information). # Prepare the database for publication as described in the [[Preparation]] tutorial. This includes the decision on whether to publish the live database or a snapshot, the creation of a table for metadata and, in certain cases, some data transformation. # Follow the [[Installation]] Tutorial to install the BioCASe Provider Software. # Set up a BioCASe Datasource and connect it to your database, then configure the database structure as described in the [[DatasourceSetup]] tutorial. # Follow the [[ABCD2Mapping]] tutorial to create a mapping for the schema you want to use. # Test and debug your BioCASe web service. Read the [[Debugging]] tutorial to learn how to do this. # Once you’re done with setting up and configuring the BioCASe Provider Software, you can register the BioCASe web service with the information network you want to publish your data to, for example with [http://www.gbif.org GBIF]. We strongly recommend to [[Special:Contact|contact the BioCASe team]] for checking your web service first as we might be able to foresee problems that are more difficult to resolve after registration. == Support == In case you run into difficulties during the installation and setup, you should * have a look at the [[FAQ]] which might list the solution for your question, * ask other BioCASe data providers for help, * get in contact with the [[Special:Contact|BioCASe team]]. Please describe exactly your environment (operating system, DBMS used), the steps that cause the problem and any error messages and hints you get. Screenshot are a big plus here. a814b7bf2e1d86609d06618d868e77e52fff8c16 0 110 963 828 2018-05-28T09:00:26Z JoergHoletschek 3 /* Voraussetzungen für den Einsatz der BioCASe Provider Software */ wikitext text/x-wiki [[http://wiki.bgbm.org/bps/index.php/BeginnersGuide This page in English]] ==Was ist die BioCASe Provider Software?== Die BioCASe Provider Software ist eine Middleware, also eine Software, die den Datenaustausch zwischen zwei voneinander entfernten Systemen erlaubt. Sie ermöglicht, Daten aus einem relationalen Datenbanksystem (z.B. einem Sammlungsverwaltungssystem) in ein Datennetzwerk einzuspeisen. Nach der Installation und Konfiguration von BioCASe sind die Daten als BioCASe Webservice im Internet verfügbar und können mittels BioCASe-Protokollanfragen abgerufen werden. Die Provider Software ist nicht abhängig von einem bestimmten Datenstandard und lässt sich mit einem beliebigen Datenschema verwenden. Die Provider Software muss auf einem Webserver des Datenlieferanten installiert werden. Kern ist der sogenannte PyWrapper, eine in Python entwickelte XML/CGI-Datenschnittstelle, die einen standardisierten Zugriff auf eine Vielzahl von Datenbankmanagementsystemen mit beliebigen Datenmodellen ermöglicht. Um diese Schnittstelle herum existieren verschiedene Tools, die die Einrichtung, Konfiguration und den Test der BioCASe-Installation und –Webservices vereinfachen. Obwohl BioCASe prinzipiell mit jedem XML-Datenschema verwendet werden kann, ist das Haupteinsatzfeld die Anbindung von biologischen Beleg- und Beobachtungsdaten an Primärdatennetzwerke zur Artenvielfalt wie dem [http://www.biocase.org BioCASe Netzwerk] und der [http://www.gbif.org Global Biodiversity Information Facility]. ==Voraussetzungen für den Einsatz der BioCASe Provider Software== Vor dem Einsatz der Provider Software sollte ein potentieller Datenlieferant drei Voraussetzungen prüfen: * Die zu veröffentlichenden Sammlungs- oder Beobachtungsdaten müssen in einem SQL-fähigen relationalen Datenbankmanagementsystem vorhanden sein oder in einem Format, zu dem ein Zugang über ODBC (Open Database Connectivity) möglich ist, wie z.B. Microsoft Excel. Nicht-relationale Datenbankmanagementsysteme werden nicht unterstützt. Derzeit kann sich die Provider Software mit den folgenden Datenbanksystemen verbinden: Microsoft Access, Microsoft SQL Server, MySQL, PostgreSQL, Oracle, Foxpro, Sybase, 4D, DB2, Firebird, Microsoft Excel. * Zur Installation der Provider Software muss ein dauerhaft mit dem Internet verbundener Webserver vorhanden sein (Apache oder Microsoft IIS). Dies kann ein Server des Datenproviders oder eines kommerziellen Hosting-Dienstes sein. Eine Veröffentlichung über eine Dial-up-Verbindung ist nicht möglich. * Auf diesem Webserver muss die Installation von Python (insofern es noch nicht installiert ist, was z.B. bei vielen gängigen Linux-Installationen der Fall ist) sowie zusätzlicher Python-Pakete möglich sein. Dies ist für Mitarbeiter der IT üblicherweise keine Hürde, solange dem keine institutionellen Richtlinien entgegenstehen. Für die optionale Erzeugung von DarwinCore-Archiven ist zusätzlich eine Java Runtime Environment nötig. BioCASe kann auch als Docker Container installiert werden, was Installation, Update und Entfernen der Provider Software sehr erleichtert. Docker ist eine Containervirtualisierungssoftware; mehr dazu im [https://docs.docker.com/get-started/ Get Started Guide]. ==Vorgehensweise== Sollte die Wahl auf die BioCASe Provider Software fallen (bei der Entscheidungsfindung hilft das [[Special:Contact|BioCASe Team]] gerne weiter), umfasst eine Anbindung von Daten üblicherweise die folgenden Schritte: # Entscheiden, welche Daten an das Netzwerk angebunden werden sollen. Dazu gehören die Klärung der Frage, wem die Daten eigentlich gehören, sowie die Festlegung der Nutzungsbedingungen/-beschränkungen und einer Lizenz. Darüber hinaus sollte geprüft werden, welche Daten '''nicht''' veröffentlicht werden sollen, weil sie gefährdete Arten betreffen. In diesem Fall können bestimmte Datensätze komplett von der Veröffentlichung ausgeschlossen werden oder bestimmte Teile davon – z.B. die Koordinaten des Fundortes – gezielt verunschärft werden. # Vorbereitung der Datenbank auf die Veröffentlichung. Dabei müssen die Entscheidung getroffen werden, ob die Live-Datenbank oder ein regelmäßig erzeugter Snapshot angebunden werden soll, und die Tabelle für die Metadaten angelegt und befüllt werden. In einigen Fällen sind weitere Datentransformationen nötig, siehe [[Preparation|Tutorial]]. # Installation der Provider Software, siehe [[Installation|Tutorial]]. # Anlegen einer BioCASe Datenquelle, Verbinden mit der Datenbank und Definition der Datenbankstruktur, siehe [[DatasourceSetup|Tutorial]]. # Anlegen eines [[ABCD2Mapping|ABCD2 Mappings]]. # Testen und Debuggen des BioCASe Webservices, siehe [[Debugging|Tutorial]]. # Nach Abschluss der Konfiguration kann der neue BioCASe-Webservice im Wunsch-Datennetzwerk angemeldet werden, z.B. bei [http://www.gbif.org GBIF]. Wir empfehlen jedoch, davor kurz das [[Special:Contact|BioCASe-Team zu kontaktieren]], das einen Blick auf den neuen Webservice werfen und auf Richtigkeit prüfen kann. Falsche Mappings lassen sich umso leichter korrigieren, je früher sie entdeckt werden. ==Support== Bei Problemen während Installation und Konfiguration empfehlen wir * In den [[FAQ]] nachzuschlagen, ob das Problem und eine Lösung dort beschrieben sind, * Befreundete Nutzer der BioCASe Provider Software um Rat zu fragen, oder * Kontakt mit dem [[Special:Contact|BioCASe Team]] aufzunehmen. Bitte dabei genau das verwendete System (verwendetes BioCASe-Version sowie Betriebs/Datenbankmanagementsystem), Fehlermeldungen und die Schritte, die dazu führten, beschreiben. Screenshots sind dabei immer von Vorteil. 2badc0d1acc9f28081c40bdf99f962a5a9d9be0e 0 6 971 966 2018-06-13T08:20:25Z JoergHoletschek 3 /* Installing BioCASe from a Docker image */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|650px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, since you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 2.5+ to run, it doesn't work with Python 3. On most Linux systems, Python 2.7 will be pre-installed, and this works fine with BioCASe. By the way: You can install several different Python versions on the same machine without any conflicts. If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. bbb5859498a9de501b087afb0d6de2ebae75c70d 0 23 972 970 2018-06-13T08:23:50Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully dunctional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool be55a57b6cb0d4b8e011f6f57138dab38fbbd9a4 973 972 2018-06-13T08:25:00Z JoergHoletschek 3 /* Version 3.7 [2018-06-12] */ wikitext text/x-wiki == Current Version 3 == === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 4588a987c9b9e6cd644d209e92e1ae0d2c9f25fb 974 973 2018-07-12T11:23:24Z JoergHoletschek 3 wikitext text/x-wiki == Current Version 3 == === Upcoming === * XML Archiving allows now to set a BioCASe filter in order to archive just a subset of the records published by a data source. === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool ab10220d9dda3f3aca0df2ca5fdbce2280ce0f4c 975 974 2018-07-27T11:37:52Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 667063eebcfae3babe50a9ffa6f7a56038412627 976 975 2018-08-16T08:54:38Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * Bugfix: Use https for Google Maps API to keep it working for installations using https. === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool a9e46fa5640570a0963193177c6357bc78cfcb22 978 976 2018-08-22T09:54:43Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * Bugfix: Use https for Google Maps API to keep it working for installations using https. * Bugfix for AnnoSys retrieval (used to cause SSL errors on certain installation). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 2ab9957c8530a0932683197f37e4777b069883ec 979 978 2018-08-22T11:08:29Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL errors on certain installation). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 23406b95d7741af2c831e4943b0fd4a5914a1eab 980 979 2018-08-23T09:32:06Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool c3a0b8445834dcd09f3957f49e996032f19608f6 981 980 2018-08-23T10:56:21Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 6a569a3a4e8885d50a8eb1e46be1496969f1f614 0 22 977 898 2018-08-16T09:24:59Z JoergHoletschek 3 wikitext text/x-wiki This page lists the most commonly used ABCD 2.06 concepts. The ABCD documentation can be found in the [http://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki], a tabular list of ABCD concepts [http://terms.tdwg.org/wiki/ABCD_2_XPath_to_Short_Name_Mapping here]. == Mandatory == List of mandatory concepts. Without them being mapped the Provider Software will not be able to create valid ABCD document: <pre> DataSets └─ DataSet ├─ ContentContacts │ └─ ContentContact │ └─ Name ├─ TechnicalContacts │ └─ TechnicalContact │ └─ Name ├─ Metadata │ ├─ Description │ │ └─ Representation │ │ ├─ @language │ │ └─ Title │ └─ RevisionData │ └─ DateModified └─ Units └─ Unit ├─ SourceInstitutionID ├─ SourceID └─ UnitID </pre> == Highly Recommended == List of recommended ABCD concepts. If you can provide this data, please do so, they're commonly used entry points to your data. '''Note:''' If you publish your data to GBIF, you are required to specify a license in the ''License'' node with the ''language'' attribute and the elements ''Text'' and ''URI''. The latter should contain a link to a well-defined license, for example https://creativecommons.org/publicdomain/zero/1.0/. More details on GBIF licensing can be found [http://www.gbif.org/terms/licences here]. <pre> DataSets └─ DataSet ├─ ContentContacts │ └─ ContentContact │ ├─ Email │ └─ Address ├─ TechnicalContacts │ └─ TechnicalContact │ ├─ Email │ └─ Address ├─ Metadata │ ├─ Description │ │ └─ Representation │ │ └─ Details │ ├─ Owners │ │ └─ Owner │ │ ├─ LogoURI │ │ ├─ URIs │ │ │ └─ URL │ │ ├─ Person │ │ │ └─ FullName │ │ └─ Organisation │ │ └─ Name │ │ └─ Representation │ │ └─ Text │ └─ IPRStatements │ └─ Licenses │ └─ License │ ├─ @language │ ├─ Text │ └─ URI └─ Units └─ Unit ├─ Identifications │ └─ Identification │ └─ Result │ ├─ @PreferredFlag │ ├─ TaxonIdentified │ │ └─ ScientificName │ │ └─ FullScientificNameString │ └─ HigherTaxa │ └─ HigherTaxon │ ├─ HigherTaxonName │ └─ HigherTaxonRank ├─ RecordBasis └─ Gathering ├─ DateTime │ └─ ISODateTimeBegin ├─ Agents │ └─ GatheringAgent │ └─ Person │ └─ FullName ├─ LocalityText ├─ Country │ ├─ ISO3166Code │ └─ Name └─ SiteCoordinateSets └─ SiteCoordinates └─ CoordinatesLatLong ├─ LongitudeDecimal ├─ LatitudeDecimal └─ CoordinateErrorDistanceInMeters </pre> == Commonly Used == List of further commonly mapped ABCD concepts. <pre> DataSets └─ DataSet ├─ Metadata │ └─ IPRStatements │ ├─ TermsOfUseStatements │ │ └─ TermsOfUse │ │ └─ Text │ ├─ Copyrights │ │ └─ Copyright │ │ └─ Text │ ├─ IPRDeclarations │ │ └─ IPRDeclaration │ │ └─ Text │ ├─ Licenses │ │ └─ License │ │ └─ Text │ ├─ Disclaimers │ │ └─ Disclaimer │ │ └─ Text │ ├─ Citations │ │ └─ Citation │ │ └─ Text │ └─ Acknowledgements │ └─ Acknowledgement │ └─ Text └─ Units └─ Unit ├─ DateLastEdited ├─ Identifications │ └─ Identification │ ├─ Date │ │ └─ ISODateTimeBegin │ └─ Identifiers │ └─ Identifier │ └─ IdentifierPersonName │ └─ FullName ├─ KindOfUnit ├─ CollectorsFieldNumber ├─ Sex ├─ Age ├─ Notes ├─ RecordURI ├─ MultiMediaObjects │ └─ MultiMediaObject │ ├─ FileURI │ ├─ Comment │ └─ Context ├─ SpecimenUnit │ └─ NomenclaturalTypeDesignations │ └─ NomenclaturalTypeDesignation │ ├─ TypeStatus │ ├─ TypifiedName │ │ └─ FullScientificNameString │ └─ NomenclaturalReference │ ├─ TitleCitation │ └─ CitationDetail └─ Gathering ├─ Project │ └─ ProjectTitle ├─ Biotope │ └─ Name ├─ Altitude │ └─ MeasurementOrFactAtomised │ ├─ LowerValue │ └─ UnitOfMeasurement ├─ Depth │ └─ MeasurementOrFactAtomised │ ├─ LowerValue │ └─ UnitOfMeasurement └─ NamedAreas └─ NamedArea ├─ AreaName └─ AreaClass </pre> <!-- {| class="wikitable" ! Category ! Element ! Discription ! Comment |- | rowspan="3" | Metadata | Title | | |- | IPR | | |- | TechnicalContact | | |- | Unit | UnitID | | |} --> cc6ecfc4757288a8c8e70771c7bf44193db9c505 983 977 2018-09-27T08:48:00Z JoergHoletschek 3 wikitext text/x-wiki This page lists the most commonly used ABCD 2.06 concepts. The ABCD documentation can be found in the [http://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki], a tabular list of ABCD concepts [http://terms.tdwg.org/wiki/ABCD_2_XPath_to_Short_Name_Mapping here]. == Mandatory == List of mandatory concepts. Without them being mapped the Provider Software will not be able to create valid ABCD document: <pre> DataSets └─ DataSet ├─ ContentContacts │ └─ ContentContact │ └─ Name ├─ TechnicalContacts │ └─ TechnicalContact │ └─ Name ├─ Metadata │ ├─ Description │ │ └─ Representation │ │ ├─ @language │ │ └─ Title │ └─ RevisionData │ └─ DateModified └─ Units └─ Unit ├─ SourceInstitutionID ├─ SourceID └─ UnitID </pre> == Highly Recommended == List of recommended ABCD concepts. If you can provide this data, please do so, they're commonly used entry points to your data. '''Note:''' If you publish your data to GBIF, you are required to specify a license in the ''License'' node with the ''language'' attribute and the elements ''Text'' and ''URI''. The latter should contain a link to a well-defined license, for example https://creativecommons.org/publicdomain/zero/1.0/. More details on GBIF licensing can be found [http://www.gbif.org/terms/licences here]. <pre> DataSets └─ DataSet ├─ ContentContacts │ └─ ContentContact │ ├─ Email │ └─ Address ├─ TechnicalContacts │ └─ TechnicalContact │ ├─ Email │ └─ Address ├─ Metadata │ ├─ Description │ │ └─ Representation │ │ └─ Details │ ├─ Owners │ │ └─ Owner │ │ ├─ LogoURI │ │ ├─ URIs │ │ │ └─ URL │ │ ├─ Person │ │ │ └─ FullName │ │ └─ Organisation │ │ └─ Name │ │ └─ Representation │ │ └─ Text │ └─ IPRStatements │ └─ Licenses │ └─ License │ ├─ @language │ ├─ Text │ └─ URI └─ Units └─ Unit ├─ Identifications │ └─ Identification │ └─ Result │ ├─ @PreferredFlag │ ├─ TaxonIdentified │ │ └─ ScientificName │ │ └─ FullScientificNameString │ └─ HigherTaxa │ └─ HigherTaxon │ ├─ HigherTaxonName │ └─ HigherTaxonRank ├─ RecordBasis └─ Gathering ├─ DateTime │ └─ ISODateTimeBegin ├─ Agents │ └─ GatheringAgent │ └─ Person │ └─ FullName ├─ LocalityText ├─ Country │ ├─ ISO3166Code │ └─ Name └─ SiteCoordinateSets └─ SiteCoordinates └─ CoordinatesLatLong ├─ LongitudeDecimal ├─ LatitudeDecimal └─ CoordinateErrorDistanceInMeters </pre> == Commonly Used == List of further commonly mapped ABCD concepts. <pre> DataSets └─ DataSet ├─ Metadata │ └─ IPRStatements │ ├─ TermsOfUseStatements │ │ └─ TermsOfUse │ │ └─ Text │ ├─ Copyrights │ │ └─ Copyright │ │ └─ Text │ ├─ IPRDeclarations │ │ └─ IPRDeclaration │ │ └─ Text │ ├─ Licenses │ │ └─ License │ │ └─ Text │ ├─ Disclaimers │ │ └─ Disclaimer │ │ └─ Text │ ├─ Citations │ │ └─ Citation │ │ └─ Text │ └─ Acknowledgements │ └─ Acknowledgement │ └─ Text └─ Units └─ Unit ├─ DateLastEdited ├─ Identifications │ └─ Identification │ ├─ Date │ │ └─ ISODateTimeBegin │ └─ Identifiers │ └─ Identifier │ └─ IdentifierPersonName │ └─ FullName ├─ KindOfUnit ├─ CollectorsFieldNumber ├─ Sex ├─ Age ├─ Notes ├─ RecordURI ├─ MultiMediaObjects │ └─ MultiMediaObject │ ├─ FileURI │ ├─ Comment │ └─ Context ├─ SpecimenUnit │ └─ NomenclaturalTypeDesignations │ └─ NomenclaturalTypeDesignation │ ├─ TypeStatus │ ├─ TypifiedName │ │ └─ FullScientificNameString │ └─ NomenclaturalReference │ ├─ TitleCitation │ └─ CitationDetail └─ Gathering ├─ Project │ └─ ProjectTitle ├─ Biotope │ └─ Name ├─ Altitude │ └─ MeasurementOrFactAtomised │ ├─ LowerValue │ └─ UnitOfMeasurement ├─ Depth │ └─ MeasurementOrFactAtomised │ ├─ LowerValue │ └─ UnitOfMeasurement └─ NamedAreas └─ NamedArea ├─ AreaName └─ AreaClass </pre> 3be3785e1920cf72c1882ed25ac3fad94d14e0d3 0 115 982 2018-09-27T08:47:09Z JoergHoletschek 3 Created page with "=Introduction= ABCD and DarwinCore both contain hundreds of elements. Some are mandatory. The curator of the DNA Bank has to define the Metadata of the DNA Bank collection fir..." wikitext text/x-wiki =Introduction= ABCD and DarwinCore both contain hundreds of elements. Some are mandatory. The curator of the DNA Bank has to define the Metadata of the DNA Bank collection first. After that the real mapping can begin. You can use [http://www.biocase.org BioCASe] or [http://www.gbif.org/ipt IPT] to provide data to GGBN, but have to make sure you use the extensions for the [http://terms.tdwg.org/wiki/GGBN_Data_Standard GGBN Data Standard]. DarwinCore and ABCD have a lot of terms in common, but use different names for the terms. Therefore the following section are splitted into ABCD and DwC. GGBN supports both standards, both have pros and cons and work well for GGBN. It depends on your data and if your institution is already providing data to GBIF which one fits better. Please contact us if you need help. <div id="wikinote"> '''Asterisk (*) means mandatory for GGBN, all others are recommended. Separate mappings for voucher specimens (mapping 1) and tissue and DNA samples (mapping 2) are preferred. This enables third parties such as GBIF or GGBN to harvest only those data that are relevant. All metadata information should be in English! Please map as many elements as possible, the better your data visible and searchable in GBIF and GGBN the better for other scientists and at the end for yourself!</div> =Mapping principles= You can provide your data to GGBN in different ways. It depends on your other collections and the way you provide your data already to GBIF which one fits best for you. GGBN can handle all of the examples below, but it might make a difference for your occurrences at GBIF. Most important is, that you provide the references between the records in a standardized way. DNA collections comprise many difference use cases. Not in all cases you will have a physicial tissue or a physicial specimen. Please contact us if you need help. ==Mapping with Specimens== A specimen in terms of GGBN and GBIF is a preserved specimen from wild origin or a living specimen (cultivated plant, captivated animal, cell culture). In case your collection or certain DNA samples have no corresponding tissue sample, you should refer to the "mother" specimen directly. <div id="wikinote">Specimens are often deposited in external collections. If you provide the proper identifiers the GGBN Data Portal can aggregate this information! We know that it is impossible to track this information back for all legacy data. But if you keep this in mind in your daily workflows for all new records and samples the quality and value of your data and collection will increase enormously.</div> {| class="wikitable" ! Mapping all at once !! Mapping two sources !! Mapping three sources |- |[[File:GGBN mapping one source.png|Thumb|300px]] || [[File:GGBN mapping two sources.png|Thumb|300px]] || [[File:GGBN mapping three sources.png|Thumb|300px]] |- |'''Pros:''' - only one mapping needed '''Cons:''' - double occurrences at GBIF, since GBIF can't handle associataions ||'''Pros:''' - Only underlying specimens records are provided, no double entries at GBIF '''Cons:''' - two mappings needed (but can be easily copied) ||'''Pros:''' - Only underlying specimens records are provided, no double entries at GBIF - Clear distinction between collections, some managers prefer this mapping type '''Cons:''' - three mappings needed (but can be easily copied) |} ==Mapping without Specimens== Often tissue samples are collected without collecting/killing the whole organism. There are two ways to handle this. {| class="wikitable" ! Mapping eVouchers !! Mapping tissues only |- |style="width: 50%;"|[[File:GGBN mapping evoucher.png|Thumb|300px]] ||[[File:GGBN mapping tissues only.png|Thumb|300px]] |- |In case you want to track an individual, sampled multiple times (e.g. vertebrates caught multiple times and marked individually, trees) you should add an Observation record as top level. This could also be accompanied by images, then called eVoucher. ||In case you don't want to track individuals, just provide your tissue records to GBIF as top level occurrence. |} ==Environmental Samples== Environmental Samples and even more environmental DNA can currently be provided to GGBN and GBIF in a very basic way only. We are working on a proposal how to map these important sample information in a way they deserve it. Have a look at our [[Use_Cases_Enviro | Environmental DNA use case]] =ABCD= ==Metadata of your DNA and tissue bank== Metadata are very important. They will be displayed with every single DNA record in the portal. Be careful with the IPR statements! {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |ContentContact |*Address |Complete Address of the responsible person |Botanic Garden and Botanical Museum Berlin-Dahlem, Freie Universität Berlin, Koenigin-Luise-Str. 6-8, 14195 Berlin, Germany |- |ContentContact |*Email |valid email address; will be used by the web portal for sending you annotations etc. |please use @ instead of masquerades like "[at]" |- |ContentContact |*Name |Person, person team responsible for curation of your DNA Bank |Gabi Droege |- |Description/Representation |*Details |Short text to describe the focus and number of samples in your DNA collection. '''It should include the following phrase:''' ''The DNA bank is part of the Global Genome Biodiversity Network (GGBN) which was founded in 2011. The network provides a technically optimized DNA collection service facility for all biological research accessible via one central web portal. The network promotes deposition of well documented reference DNA samples after project completion or data publication from scientists of other universities and institutions.'' |The DNA bank of the Botanic Garden and Botanical Museum Berlin-Dahlem holds currently a collection of 20000 plant DNA and tissue samples growing constantly. Its core collection focuses on the flora of Germany but it comprises botanical samples collected worldwide. |- |Description/Representation |*@language |language of description |should be "en" as literal |- |Description/Representation |*Title |Short title that describes your DNA collection |DNA Bank of the Herbarium Berolinense |- |IPRStatements/Citation |*Text | |Droege, G. (Ed.) 2008 - (continuously updated): DNA samples of the DNA bank at the BGBM (Botanic Garden and Botanical Museum Berlin-Dahlem). |- |IPRStatements/Citation |*@language |language of citation |should be "en" as literal |- |IPRStatements/Copyright |*Text | |The copyright for any material created by the DNA bank of the BGBM is reserved. The duplication or use of information and data such as texts or images is only permitted with the indication of the source or with prior approval by the BGBM. |- |IPRStatements/Copyright |*@language |language of copyright |should be "en" as literal |- |IPRStatements/TermsOfUse |*Text | |The use of the data is allowed only for non-profit scientific use and for non-profit nature conservation purpose. The data bases or part of it may only be used or copied by the written permission from the legal owner. |- |IPRStatements/TermsOfUse |*@language |language of the terms of use |should be "en" as literal |- | |IconURI |complete url path to the logo of your institution | |- |Owner |*Address |Complete Address of the institution that owns the DNA bank samples and data | |- |Owner/Representation |*Text |name of your institution |Botanic Garden an Botanical Museum Berlin-Dahlem (Freie Universität Berlin) |- |Owner |*URL |path to website of your institution |http://www.bgbm.org |- |RevisionData |*DateModified |date of last modification of your data | |- |TechnicalContact |*Address |Complete Address of the responsible person |Botanic Garden and Botanical Museum Berlin-Dahlem, Freie Universität Berlin, Koenigin-Luise-Str. 6-8, 14195 Berlin, Germany |- |TechnicalContact |*Email |valid email address; will be used by the web portal for sending you annotations etc. |please use @ instead of masquerades like "[at]" |- |TechnicalContact |*Name |Person, person team responsible for technical issues of your database |Gabriele Droege |- |Unit |*RecordBasis |this value must be part of the ABCD vocabulary |must be "MaterialSample" |- |Unit |*KindOfUnit |the type of sample |e.g. DNA |} ==Sample data== ===The sample identifiers=== We use identifier the same way as GBIF does. In addition to the traditional triple ID (see below) you can also provide the GUID. '''Please note that UnitID should be unique in one database!''' Mostly the extraction number is used for it. {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Unit |*<span style="color:red">SourceID</span> |short description of relevant collection |should be "DNA Bank" or "tissue collection" |- |Unit |*<span style="color:red">SourceInstitutionID</span> |short name/abbreviation of relevant institution |BGBM |- |Unit |*<span style="color:red">UnitID</span> |DNA extraction number or tissue number |DNA 123 |- |Unit |UnitGUID |sample record GUID |f6c7fd5a-2f04-4dcb-ba38-0230bf196d30 |} ===GGBN terms=== The following fields are highly recommended, some are mandatory (marked with *), but feel free to map more! <div id="wikinote">'''Please have a look at the [http://terms.tdwg.org/wiki/GGBN_Data_Standard GGBN Data Standard] documentation where all GGBN terms are described in detail. The table below is just listing the most important ones. Since the Nagoya Protocol has come into force we higly recommend to map the Permit fields. They will become mandatory for providing data to GGBN at the end of 2020. Until then all core members must have implemented the required fields in their databases.'''</div> {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- | |*RecordBasis |The specific nature of the data record, controlled vocabulary! |"PreservedSpecimen", "MaterialSample", "FossilSpecimen", "LivingSpecimen", "HumanObservation", "MachineObservation"; for Tissue or DNA samples use "MaterialSample" |- | |*materialSampleType |Classification of kind of physical sample in addition to BasisOfRecord/RecordBasis and Preparation Type |"tissue", "DNA", "specimen" |- |SpecimenUnit/Preparation |*preparationType |description of type of material, free text |or DNA: gDNA, eDNA, aDNA; for tissues/specimens: leaf, muscle, leg, blood |- |SpecimenUnit/Preparation |*preparationDate |date of DNA extraction |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparatioMaterials |extraction kit or protocoll |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparedBy |extraction staff |if unknown type "unknown" |- |SpecimenUnit/Preservation |preservation |preservation of the tissue or DNA |if unknown type "unknown" |- |SpecimenUnit/Loan/Permit |*permitStatus |Information about the presence, absence or other basic status of permits associated with the sample(s), controlled vocabulary! |Permit available, Permit not required, Permit not available, '''Unknown Material collected after 2014-10-12 cannot be in "Unknown" permit status!''' |- |SpecimenUnit/Loan/Permit |*permitStatusQualifier |Description of why a certain permit was not required or why Permit Status is unknown |"no national requirement for a permit at date of access", "officially authorized illegal holder", "collected on private land", "pre-Nagoya" |- |SpecimenUnit/Loan/Permit |*permitType |A permit is a document that allows someone to take an action that otherwise would not be allowed, controlled vocabulary! |Collecting Permit, Import Permit, Export Permit, Intellectual Property Rights, Copyright, Patent, Data use, Phytosanitary, Salvage, Exemption Permit, Material Transfer Agreement, Internationally Recognized Certificate of Compliance, Contract, Memorandum of Understanding, Memorandum of Cooperation, Veterinary Certificate, Human Pathogens, Genetically Modified Organism, Other |- |SpecimenUnit/Loan/Permit |*permitText |The text of a permit related to the gathering/shipping or further details | |- |GGBN/Amplification |amplificationDate |date of amplification; if unknown or general without content you don't have to map it |should be ISO format yyyy-mm-dd |- |GGBN/Amplification |marker | |COX1 |- |GGBN/Amplification |geneticAccessionNumber |the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD; this is a repeatable element, you can provide as much as you want |e.g. AJ45567 |- |GGBN/Amplification |genBankNumber-URI |complete link to the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD | |- |SpecimenUnit |blockedUntil |in case the sample is blocked until a specific date it's nevertheless searchable but customers cannot order it |ISO format |- |SpecimenUnit |blocked |sample is blocked (e.g. because it is consumed), but data are still available |Yes/No |- |GGBN |concentration |concentration of the DNA |65,34 |- |GGBN |@unit |unit of the concentration |µg/ml |- |GGBN |ratioOfAbsorbance260_230 |map only if filled with content |1,2 |- |GGBN |ratioOfAbsorbance260_280 |map only if filled with content |1,8 |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===The voucher identifiers=== {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Associations/UnitAssociation |*UnitID |the UnitID or CatalogueNumber used for GBIF |e.g. the barcode number of your specimens |- |Associations/UnitAssociation |*SourceInstitutionID |the SourceInstitutionID or InstitutionCode used for GBIF |e.g. the acronym of your institution |- |Associations/UnitAssociation |*SourceName |the SourceID or CollectionCode used for GBIF |e.g. the name of the collection where the specimen belongs to, e.g. "Birds" |- |Associations/UnitAssociation |*AssociationType |the Relation between the DNA and the voucher |e.g. "DNA and voucher from same individual" |- |Associations/UnitAssociation |*DatasetAccessPoint |the wrapper url of the voucher record |e.g. "http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar" |} ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} ===Multimedia items of the voucher=== These should be mapped in the specimen mapping, not the DNA mapping. =Darwin Core= We recommend using [http://www.gbif.org/ipt/ IPT] for providing data as Darwin Core-Archive. '''Mandatory for GGBN:''' Select '''occurrence''' as core and add '''GGBN Material Sample''' and '''Darwin Core Resource Relationship''' as extensions. ==Metadata of your DNA and tissue bank== Please follow the example in the IPT documentation: https://github.com/gbif/ipt/wiki/IPT2ManualNotes.wiki#basic-metadata '''Note:''' The description of your dataset should contain the following phrase: ''The DNA bank is part of the Global Genome Biodiversity Network (GGBN) which was founded in 2011. The network provides a technically optimized DNA collection service facility for all biological research accessible via one central web portal. The network promotes deposition of well documented reference DNA samples after project completion or data publication from scientists of other universities and institutions.'' ==Sample data== ===The sample identifiers=== We use identifier the same way as GBIF does. In addition to the traditional triple ID (see below) you can also provide the GUID. '''Please note that CatalogNumber should be unique in one database!''' Mostly the extraction number is used for it. {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Occurrence |*<span style="color:red">institutionCode</span> |short description of relevant collection |should be "DNA Bank" or "tissue collection" |- |Occurrence |*<span style="color:red">collectionCode</span> |short name/abbreviation of relevant institution |NMNH |- |Occurrence |*<span style="color:red">catalogNumber</span> |DNA extraction number or tissue number |DNA 123 |- |Occurrence |occurrenceID |sample record GUID | http://n2t.net/ark:/65665/304ed89be-b1ed-4e71-b210-dbbddfadb776 |} ===GGBN terms=== The following fields are highly recommended, some are mandatory (marked with *), but feel free to map more! <div id="wikinote">'''Please have a look at the [http://terms.tdwg.org/wiki/GGBN_Data_Standard GGBN Data Standard] documentation where all GGBN terms are described in detail. The table below is just listing the most important ones. Since the Nagoya Protocol has come into force we higly recommend to map the Permit fields. They will become mandatory for providing data to GGBN at the end of 2020. Until then all core members must have implemented the required fields in their databases.'''</div> {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Occurrence Core |*basisOfRecord |The specific nature of the data record, controlled vocabulary! |"PreservedSpecimen", "MaterialSample", "FossilSpecimen", "LivingSpecimen", "HumanObservation", "MachineObservation"; for Tissue or DNA samples use "MaterialSample" |- |Material Sample Extension |*materialSampleType |Classification of kind of physical sample in addition to BasisOfRecord/RecordBasis and Preparation Type |"tissue", "DNA", "specimen" |- |Preparation Extension |*preparationType |description of type of material, free text |or DNA: gDNA, eDNA, aDNA; for tissues/specimens: leaf, muscle, leg, blood |- |Preparation Extension |*preparationDate |date of DNA extraction |if unknown type "unknown" |- |Preparation Extension |*preparatioMaterials |extraction kit or protocoll |if unknown type "unknown" |- |Preparation Extension |*preparationStaff |extraction staff |if unknown type "unknown" |- |Preservation Extension |preservation |preservation of the tissue or DNA |if unknown type "unknown" |- |Permit Extension |*permitStatus |Information about the presence, absence or other basic status of permits associated with the sample(s), controlled vocabulary! |Permit available, Permit not required, Permit not available, '''Unknown Material collected after 2014-10-12 cannot be in "Unknown" permit status!''' |- |Permit Extension |*permitStatusQualifier |Description of why a certain permit was not required or why Permit Status is unknown |"no national requirement for a permit at date of access", "officially authorized illegal holder", "collected on private land", "pre-Nagoya" |- |Permit Extension |*permitType |A permit is a document that allows someone to take an action that otherwise would not be allowed, controlled vocabulary! |Collecting Permit, Import Permit, Export Permit, Intellectual Property Rights, Copyright, Patent, Data use, Phytosanitary, Salvage, Exemption Permit, Material Transfer Agreement, Internationally Recognized Certificate of Compliance, Contract, Memorandum of Understanding, Memorandum of Cooperation, Veterinary Certificate, Human Pathogens, Genetically Modified Organism, Other |- |Permit Extension |*permitText |The text of a permit related to the gathering/shipping or further details | |- |Amplification Extension |amplificationDate |date of amplification; if unknown or general without content you don't have to map it |should be ISO format yyyy-mm-dd |- |Amplification Extension |marker | |COX1 |- |Amplification Extension |geneticAccessionNumber |the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD; this is a repeatable element, you can provide as much as you want |e.g. AJ45567 |- |Amplification Extension |genBankNumber-URI |complete link to the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD | |- |Loan Extension |blockedUntil |in case the sample is blocked until a specific date it's nevertheless searchable but customers cannot order it |ISO format |- |Loan Extension |blocked |sample is blocked (e.g. because it is consumed), but data are still available |Yes/No |- |Material Sample Extension |concentration |concentration of the DNA |65,34 |- |Material Sample Extension |@unit |unit of the concentration |µg/ml |- |Material Sample Extension |ratioOfAbsorbance260_230 |map only if filled with content |1,2 |- |Material Sample Extension |ratioOfAbsorbance260_280 |map only if filled with content |1,8 |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===The voucher identifiers=== Please use the Darwin Core Resource Relationship Class {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Resource Relationship |*resourceRelationshipID |concatenated string with the triple id used for GBIF plus the accesspoint of the IPT archive providing this record and the guid (the latter one is not mandatory, but recommended) |e.g. catalogNumber=11718653&collectionCode=Botany&institutionCode=US&accesspoint=http://collections.mnh.si.edu/ipt/archive.do?r=nmnhdwca&guid=http://n2t.net/ark:/65665/303e7eecb-4e8d-4fce-a251-6c8fa3f2863d |- | Resource Relationship |*relationshipOfResource |the Relation between the DNA and the voucher or the tissue and the voucher |e.g. " same individual", “same population”, “same ex situ individual” |} ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Occurrence Core |*recordNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- | Occurrence Core |*recordedBy |the Collector or Collector Team |e.g. Scholz & Sipman |- | Occurrence Core |*minimumElevationInMeters |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- | Occurrence Core |maximumElevationInMeters | |e.g. 200 |- |Occurrence Core |*countryCode |ISO code of the country where the voucher was collected |e.g. US |- | Occurrence Core |*country |english Name of the country |e.g. United States of America |- | Occurrence Core |*eventDate |date when voucher was collected, if you have content you can also use ISO format |e.g. 21. April 1951 |- | Occurrence Core |*locality | |e.g. 5km NO Berlin |- | Occurrence Core |continent |name of continent |e.g. Europe |- | Occurrence Core |*decimalLatitude | |e.g. -15,88876 |- | Occurrence Core |*decimalLongitude | |e.g. 72,88876 |} ===Scientific Name of the voucher=== {| class="wikitable" ! | Group !Element !Remarks !Example |- |Occurrence Core |*Family |Please provide at least one higher taxon, usually family |e.g. Asteraceae, Paridae |- |Occurrence Core |*scientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Occurrence Core |*specificEpithet | |e.g. factorovskyi |- |Occurrence Core |*genus | |e.g. Aaronsohnia |- |Occurrence Core |*infraspecificEpithet | |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*taxonRank | |e.g. var. |} ===Multimedia items of the voucher=== These should be mapped in the specimen mapping, not the DNA/tissue mapping. If you have histological images you should map them here. a2a7dd31ce1a677ee0c34a99982eda504ae0c3ec 984 982 2018-09-27T08:48:17Z JoergHoletschek 3 wikitext text/x-wiki =Introduction= ABCD and DarwinCore both contain hundreds of elements. Some are mandatory. The curator of the DNA Bank has to define the Metadata of the DNA Bank collection first. After that the real mapping can begin. You can use [http://www.biocase.org BioCASe] or [http://www.gbif.org/ipt IPT] to provide data to GGBN, but have to make sure you use the extensions for the [http://terms.tdwg.org/wiki/GGBN_Data_Standard GGBN Data Standard]. DarwinCore and ABCD have a lot of terms in common, but use different names for the terms. Therefore the following section are splitted into ABCD and DwC. GGBN supports both standards, both have pros and cons and work well for GGBN. It depends on your data and if your institution is already providing data to GBIF which one fits better. Please contact us if you need help. <div id="wikinote"> '''Asterisk (*) means mandatory for GGBN, all others are recommended. Separate mappings for voucher specimens (mapping 1) and tissue and DNA samples (mapping 2) are preferred. This enables third parties such as GBIF or GGBN to harvest only those data that are relevant. All metadata information should be in English! Please map as many elements as possible, the better your data visible and searchable in GBIF and GGBN the better for other scientists and at the end for yourself!</div> =Mapping principles= You can provide your data to GGBN in different ways. It depends on your other collections and the way you provide your data already to GBIF which one fits best for you. GGBN can handle all of the examples below, but it might make a difference for your occurrences at GBIF. Most important is, that you provide the references between the records in a standardized way. DNA collections comprise many difference use cases. Not in all cases you will have a physicial tissue or a physicial specimen. Please contact us if you need help. ==Mapping with Specimens== A specimen in terms of GGBN and GBIF is a preserved specimen from wild origin or a living specimen (cultivated plant, captivated animal, cell culture). In case your collection or certain DNA samples have no corresponding tissue sample, you should refer to the "mother" specimen directly. <div id="wikinote">Specimens are often deposited in external collections. If you provide the proper identifiers the GGBN Data Portal can aggregate this information! We know that it is impossible to track this information back for all legacy data. But if you keep this in mind in your daily workflows for all new records and samples the quality and value of your data and collection will increase enormously.</div> {| class="wikitable" ! Mapping all at once !! Mapping two sources !! Mapping three sources |- |[[File:GGBN mapping one source.png|Thumb|300px]] || [[File:GGBN mapping two sources.png|Thumb|300px]] || [[File:GGBN mapping three sources.png|Thumb|300px]] |- |'''Pros:''' - only one mapping needed '''Cons:''' - double occurrences at GBIF, since GBIF can't handle associataions ||'''Pros:''' - Only underlying specimens records are provided, no double entries at GBIF '''Cons:''' - two mappings needed (but can be easily copied) ||'''Pros:''' - Only underlying specimens records are provided, no double entries at GBIF - Clear distinction between collections, some managers prefer this mapping type '''Cons:''' - three mappings needed (but can be easily copied) |} ==Mapping without Specimens== Often tissue samples are collected without collecting/killing the whole organism. There are two ways to handle this. {| class="wikitable" ! Mapping eVouchers !! Mapping tissues only |- |style="width: 50%;"|[[File:GGBN mapping evoucher.png|Thumb|300px]] ||[[File:GGBN mapping tissues only.png|Thumb|300px]] |- |In case you want to track an individual, sampled multiple times (e.g. vertebrates caught multiple times and marked individually, trees) you should add an Observation record as top level. This could also be accompanied by images, then called eVoucher. ||In case you don't want to track individuals, just provide your tissue records to GBIF as top level occurrence. |} ==Environmental Samples== Environmental Samples and even more environmental DNA can currently be provided to GGBN and GBIF in a very basic way only. We are working on a proposal how to map these important sample information in a way they deserve it. Have a look at our [[Use_Cases_Enviro | Environmental DNA use case]] =ABCD= ==Metadata of your DNA and tissue bank== Metadata are very important. They will be displayed with every single DNA record in the portal. Be careful with the IPR statements! {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |ContentContact |*Address |Complete Address of the responsible person |Botanic Garden and Botanical Museum Berlin-Dahlem, Freie Universität Berlin, Koenigin-Luise-Str. 6-8, 14195 Berlin, Germany |- |ContentContact |*Email |valid email address; will be used by the web portal for sending you annotations etc. |please use @ instead of masquerades like "[at]" |- |ContentContact |*Name |Person, person team responsible for curation of your DNA Bank |Gabi Droege |- |Description/Representation |*Details |Short text to describe the focus and number of samples in your DNA collection. '''It should include the following phrase:''' ''The DNA bank is part of the Global Genome Biodiversity Network (GGBN) which was founded in 2011. The network provides a technically optimized DNA collection service facility for all biological research accessible via one central web portal. The network promotes deposition of well documented reference DNA samples after project completion or data publication from scientists of other universities and institutions.'' |The DNA bank of the Botanic Garden and Botanical Museum Berlin-Dahlem holds currently a collection of 20000 plant DNA and tissue samples growing constantly. Its core collection focuses on the flora of Germany but it comprises botanical samples collected worldwide. |- |Description/Representation |*@language |language of description |should be "en" as literal |- |Description/Representation |*Title |Short title that describes your DNA collection |DNA Bank of the Herbarium Berolinense |- |IPRStatements/Citation |*Text | |Droege, G. (Ed.) 2008 - (continuously updated): DNA samples of the DNA bank at the BGBM (Botanic Garden and Botanical Museum Berlin-Dahlem). |- |IPRStatements/Citation |*@language |language of citation |should be "en" as literal |- |IPRStatements/Copyright |*Text | |The copyright for any material created by the DNA bank of the BGBM is reserved. The duplication or use of information and data such as texts or images is only permitted with the indication of the source or with prior approval by the BGBM. |- |IPRStatements/Copyright |*@language |language of copyright |should be "en" as literal |- |IPRStatements/TermsOfUse |*Text | |The use of the data is allowed only for non-profit scientific use and for non-profit nature conservation purpose. The data bases or part of it may only be used or copied by the written permission from the legal owner. |- |IPRStatements/TermsOfUse |*@language |language of the terms of use |should be "en" as literal |- | |IconURI |complete url path to the logo of your institution | |- |Owner |*Address |Complete Address of the institution that owns the DNA bank samples and data | |- |Owner/Representation |*Text |name of your institution |Botanic Garden an Botanical Museum Berlin-Dahlem (Freie Universität Berlin) |- |Owner |*URL |path to website of your institution |http://www.bgbm.org |- |RevisionData |*DateModified |date of last modification of your data | |- |TechnicalContact |*Address |Complete Address of the responsible person |Botanic Garden and Botanical Museum Berlin-Dahlem, Freie Universität Berlin, Koenigin-Luise-Str. 6-8, 14195 Berlin, Germany |- |TechnicalContact |*Email |valid email address; will be used by the web portal for sending you annotations etc. |please use @ instead of masquerades like "[at]" |- |TechnicalContact |*Name |Person, person team responsible for technical issues of your database |Gabriele Droege |- |Unit |*RecordBasis |this value must be part of the ABCD vocabulary |must be "MaterialSample" |- |Unit |*KindOfUnit |the type of sample |e.g. DNA |} ==Sample data== ===The sample identifiers=== We use identifier the same way as GBIF does. In addition to the traditional triple ID (see below) you can also provide the GUID. '''Please note that UnitID should be unique in one database!''' Mostly the extraction number is used for it. {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Unit |*<span style="color:red">SourceID</span> |short description of relevant collection |should be "DNA Bank" or "tissue collection" |- |Unit |*<span style="color:red">SourceInstitutionID</span> |short name/abbreviation of relevant institution |BGBM |- |Unit |*<span style="color:red">UnitID</span> |DNA extraction number or tissue number |DNA 123 |- |Unit |UnitGUID |sample record GUID |f6c7fd5a-2f04-4dcb-ba38-0230bf196d30 |} ===GGBN terms=== The following fields are highly recommended, some are mandatory (marked with *), but feel free to map more! <div id="wikinote">'''Please have a look at the [http://terms.tdwg.org/wiki/GGBN_Data_Standard GGBN Data Standard] documentation where all GGBN terms are described in detail. The table below is just listing the most important ones. Since the Nagoya Protocol has come into force we higly recommend to map the Permit fields. They will become mandatory for providing data to GGBN at the end of 2020. Until then all core members must have implemented the required fields in their databases.'''</div> {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- | |*RecordBasis |The specific nature of the data record, controlled vocabulary! |"PreservedSpecimen", "MaterialSample", "FossilSpecimen", "LivingSpecimen", "HumanObservation", "MachineObservation"; for Tissue or DNA samples use "MaterialSample" |- | |*materialSampleType |Classification of kind of physical sample in addition to BasisOfRecord/RecordBasis and Preparation Type |"tissue", "DNA", "specimen" |- |SpecimenUnit/Preparation |*preparationType |description of type of material, free text |or DNA: gDNA, eDNA, aDNA; for tissues/specimens: leaf, muscle, leg, blood |- |SpecimenUnit/Preparation |*preparationDate |date of DNA extraction |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparatioMaterials |extraction kit or protocoll |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparedBy |extraction staff |if unknown type "unknown" |- |SpecimenUnit/Preservation |preservation |preservation of the tissue or DNA |if unknown type "unknown" |- |SpecimenUnit/Loan/Permit |*permitStatus |Information about the presence, absence or other basic status of permits associated with the sample(s), controlled vocabulary! |Permit available, Permit not required, Permit not available, '''Unknown Material collected after 2014-10-12 cannot be in "Unknown" permit status!''' |- |SpecimenUnit/Loan/Permit |*permitStatusQualifier |Description of why a certain permit was not required or why Permit Status is unknown |"no national requirement for a permit at date of access", "officially authorized illegal holder", "collected on private land", "pre-Nagoya" |- |SpecimenUnit/Loan/Permit |*permitType |A permit is a document that allows someone to take an action that otherwise would not be allowed, controlled vocabulary! |Collecting Permit, Import Permit, Export Permit, Intellectual Property Rights, Copyright, Patent, Data use, Phytosanitary, Salvage, Exemption Permit, Material Transfer Agreement, Internationally Recognized Certificate of Compliance, Contract, Memorandum of Understanding, Memorandum of Cooperation, Veterinary Certificate, Human Pathogens, Genetically Modified Organism, Other |- |SpecimenUnit/Loan/Permit |*permitText |The text of a permit related to the gathering/shipping or further details | |- |GGBN/Amplification |amplificationDate |date of amplification; if unknown or general without content you don't have to map it |should be ISO format yyyy-mm-dd |- |GGBN/Amplification |marker | |COX1 |- |GGBN/Amplification |geneticAccessionNumber |the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD; this is a repeatable element, you can provide as much as you want |e.g. AJ45567 |- |GGBN/Amplification |genBankNumber-URI |complete link to the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD | |- |SpecimenUnit |blockedUntil |in case the sample is blocked until a specific date it's nevertheless searchable but customers cannot order it |ISO format |- |SpecimenUnit |blocked |sample is blocked (e.g. because it is consumed), but data are still available |Yes/No |- |GGBN |concentration |concentration of the DNA |65,34 |- |GGBN |@unit |unit of the concentration |µg/ml |- |GGBN |ratioOfAbsorbance260_230 |map only if filled with content |1,2 |- |GGBN |ratioOfAbsorbance260_280 |map only if filled with content |1,8 |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===The voucher identifiers=== {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Associations/UnitAssociation |*UnitID |the UnitID or CatalogueNumber used for GBIF |e.g. the barcode number of your specimens |- |Associations/UnitAssociation |*SourceInstitutionID |the SourceInstitutionID or InstitutionCode used for GBIF |e.g. the acronym of your institution |- |Associations/UnitAssociation |*SourceName |the SourceID or CollectionCode used for GBIF |e.g. the name of the collection where the specimen belongs to, e.g. "Birds" |- |Associations/UnitAssociation |*AssociationType |the Relation between the DNA and the voucher |e.g. "DNA and voucher from same individual" |- |Associations/UnitAssociation |*DatasetAccessPoint |the wrapper url of the voucher record |e.g. "http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar" |} ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} ===Multimedia items of the voucher=== These should be mapped in the specimen mapping, not the DNA mapping. =Darwin Core= We recommend using [http://www.gbif.org/ipt/ IPT] for providing data as Darwin Core-Archive. '''Mandatory for GGBN:''' Select '''occurrence''' as core and add '''GGBN Material Sample''' and '''Darwin Core Resource Relationship''' as extensions. ==Metadata of your DNA and tissue bank== Please follow the example in the IPT documentation: https://github.com/gbif/ipt/wiki/IPT2ManualNotes.wiki#basic-metadata '''Note:''' The description of your dataset should contain the following phrase: ''The DNA bank is part of the Global Genome Biodiversity Network (GGBN) which was founded in 2011. The network provides a technically optimized DNA collection service facility for all biological research accessible via one central web portal. The network promotes deposition of well documented reference DNA samples after project completion or data publication from scientists of other universities and institutions.'' ==Sample data== ===The sample identifiers=== We use identifier the same way as GBIF does. In addition to the traditional triple ID (see below) you can also provide the GUID. '''Please note that CatalogNumber should be unique in one database!''' Mostly the extraction number is used for it. {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Occurrence |*<span style="color:red">institutionCode</span> |short description of relevant collection |should be "DNA Bank" or "tissue collection" |- |Occurrence |*<span style="color:red">collectionCode</span> |short name/abbreviation of relevant institution |NMNH |- |Occurrence |*<span style="color:red">catalogNumber</span> |DNA extraction number or tissue number |DNA 123 |- |Occurrence |occurrenceID |sample record GUID | http://n2t.net/ark:/65665/304ed89be-b1ed-4e71-b210-dbbddfadb776 |} ===GGBN terms=== The following fields are highly recommended, some are mandatory (marked with *), but feel free to map more! <div id="wikinote">'''Please have a look at the [http://terms.tdwg.org/wiki/GGBN_Data_Standard GGBN Data Standard] documentation where all GGBN terms are described in detail. The table below is just listing the most important ones. Since the Nagoya Protocol has come into force we higly recommend to map the Permit fields. They will become mandatory for providing data to GGBN at the end of 2020. Until then all core members must have implemented the required fields in their databases.'''</div> {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Occurrence Core |*basisOfRecord |The specific nature of the data record, controlled vocabulary! |"PreservedSpecimen", "MaterialSample", "FossilSpecimen", "LivingSpecimen", "HumanObservation", "MachineObservation"; for Tissue or DNA samples use "MaterialSample" |- |Material Sample Extension |*materialSampleType |Classification of kind of physical sample in addition to BasisOfRecord/RecordBasis and Preparation Type |"tissue", "DNA", "specimen" |- |Preparation Extension |*preparationType |description of type of material, free text |or DNA: gDNA, eDNA, aDNA; for tissues/specimens: leaf, muscle, leg, blood |- |Preparation Extension |*preparationDate |date of DNA extraction |if unknown type "unknown" |- |Preparation Extension |*preparatioMaterials |extraction kit or protocoll |if unknown type "unknown" |- |Preparation Extension |*preparationStaff |extraction staff |if unknown type "unknown" |- |Preservation Extension |preservation |preservation of the tissue or DNA |if unknown type "unknown" |- |Permit Extension |*permitStatus |Information about the presence, absence or other basic status of permits associated with the sample(s), controlled vocabulary! |Permit available, Permit not required, Permit not available, '''Unknown Material collected after 2014-10-12 cannot be in "Unknown" permit status!''' |- |Permit Extension |*permitStatusQualifier |Description of why a certain permit was not required or why Permit Status is unknown |"no national requirement for a permit at date of access", "officially authorized illegal holder", "collected on private land", "pre-Nagoya" |- |Permit Extension |*permitType |A permit is a document that allows someone to take an action that otherwise would not be allowed, controlled vocabulary! |Collecting Permit, Import Permit, Export Permit, Intellectual Property Rights, Copyright, Patent, Data use, Phytosanitary, Salvage, Exemption Permit, Material Transfer Agreement, Internationally Recognized Certificate of Compliance, Contract, Memorandum of Understanding, Memorandum of Cooperation, Veterinary Certificate, Human Pathogens, Genetically Modified Organism, Other |- |Permit Extension |*permitText |The text of a permit related to the gathering/shipping or further details | |- |Amplification Extension |amplificationDate |date of amplification; if unknown or general without content you don't have to map it |should be ISO format yyyy-mm-dd |- |Amplification Extension |marker | |COX1 |- |Amplification Extension |geneticAccessionNumber |the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD; this is a repeatable element, you can provide as much as you want |e.g. AJ45567 |- |Amplification Extension |genBankNumber-URI |complete link to the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD | |- |Loan Extension |blockedUntil |in case the sample is blocked until a specific date it's nevertheless searchable but customers cannot order it |ISO format |- |Loan Extension |blocked |sample is blocked (e.g. because it is consumed), but data are still available |Yes/No |- |Material Sample Extension |concentration |concentration of the DNA |65,34 |- |Material Sample Extension |@unit |unit of the concentration |µg/ml |- |Material Sample Extension |ratioOfAbsorbance260_230 |map only if filled with content |1,2 |- |Material Sample Extension |ratioOfAbsorbance260_280 |map only if filled with content |1,8 |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===The voucher identifiers=== Please use the Darwin Core Resource Relationship Class {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Resource Relationship |*resourceRelationshipID |concatenated string with the triple id used for GBIF plus the accesspoint of the IPT archive providing this record and the guid (the latter one is not mandatory, but recommended) |e.g. catalogNumber=11718653&collectionCode=Botany&institutionCode=US&accesspoint=http://collections.mnh.si.edu/ipt/archive.do?r=nmnhdwca&guid=http://n2t.net/ark:/65665/303e7eecb-4e8d-4fce-a251-6c8fa3f2863d |- | Resource Relationship |*relationshipOfResource |the Relation between the DNA and the voucher or the tissue and the voucher |e.g. " same individual", “same population”, “same ex situ individual” |} ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Occurrence Core |*recordNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- | Occurrence Core |*recordedBy |the Collector or Collector Team |e.g. Scholz & Sipman |- | Occurrence Core |*minimumElevationInMeters |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- | Occurrence Core |maximumElevationInMeters | |e.g. 200 |- |Occurrence Core |*countryCode |ISO code of the country where the voucher was collected |e.g. US |- | Occurrence Core |*country |english Name of the country |e.g. United States of America |- | Occurrence Core |*eventDate |date when voucher was collected, if you have content you can also use ISO format |e.g. 21. April 1951 |- | Occurrence Core |*locality | |e.g. 5km NO Berlin |- | Occurrence Core |continent |name of continent |e.g. Europe |- | Occurrence Core |*decimalLatitude | |e.g. -15,88876 |- | Occurrence Core |*decimalLongitude | |e.g. 72,88876 |} ===Scientific Name of the voucher=== {| class="wikitable" ! | Group !Element !Remarks !Example |- |Occurrence Core |*Family |Please provide at least one higher taxon, usually family |e.g. Asteraceae, Paridae |- |Occurrence Core |*scientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Occurrence Core |*specificEpithet | |e.g. factorovskyi |- |Occurrence Core |*genus | |e.g. Aaronsohnia |- |Occurrence Core |*infraspecificEpithet | |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*taxonRank | |e.g. var. |} ===Multimedia items of the voucher=== These should be mapped in the specimen mapping, not the DNA/tissue mapping. If you have histological images you should map them here. <!-- {| class="wikitable" ! Category ! Element ! Discription ! Comment |- | rowspan="3" | Metadata | Title | | |- | IPR | | |- | TechnicalContact | | |- | Unit | UnitID | | |} --> ea40c43b02d9737a4fc33cd200743b443777c137 985 984 2018-09-27T08:50:41Z JoergHoletschek 3 wikitext text/x-wiki =Introduction= ABCD and DarwinCore both contain hundreds of elements. Some are mandatory. The curator of the DNA Bank has to define the Metadata of the DNA Bank collection first. After that the real mapping can begin. You can use [http://www.biocase.org BioCASe] or [http://www.gbif.org/ipt IPT] to provide data to GGBN, but have to make sure you use the extensions for the [http://terms.tdwg.org/wiki/GGBN_Data_Standard GGBN Data Standard]. DarwinCore and ABCD have a lot of terms in common, but use different names for the terms. Therefore the following section are splitted into ABCD and DwC. GGBN supports both standards, both have pros and cons and work well for GGBN. It depends on your data and if your institution is already providing data to GBIF which one fits better. Please contact us if you need help. <div id="wikinote"> '''Asterisk (*) means mandatory for GGBN, all others are recommended. Separate mappings for voucher specimens (mapping 1) and tissue and DNA samples (mapping 2) are preferred. This enables third parties such as GBIF or GGBN to harvest only those data that are relevant. All metadata information should be in English! Please map as many elements as possible, the better your data visible and searchable in GBIF and GGBN the better for other scientists and at the end for yourself!</div> =Mapping principles= You can provide your data to GGBN in different ways. It depends on your other collections and the way you provide your data already to GBIF which one fits best for you. GGBN can handle all of the examples below, but it might make a difference for your occurrences at GBIF. Most important is, that you provide the references between the records in a standardized way. DNA collections comprise many difference use cases. Not in all cases you will have a physicial tissue or a physicial specimen. Please contact us if you need help. ==Mapping with Specimens== A specimen in terms of GGBN and GBIF is a preserved specimen from wild origin or a living specimen (cultivated plant, captivated animal, cell culture). In case your collection or certain DNA samples have no corresponding tissue sample, you should refer to the "mother" specimen directly. <div id="wikinote">Specimens are often deposited in external collections. If you provide the proper identifiers the GGBN Data Portal can aggregate this information! We know that it is impossible to track this information back for all legacy data. But if you keep this in mind in your daily workflows for all new records and samples the quality and value of your data and collection will increase enormously.</div> {| class="wikitable" ! Mapping all at once !! Mapping two sources !! Mapping three sources |- |[[File:GGBN mapping one source.png|Thumb|300px]] || [[File:GGBN mapping two sources.png|Thumb|300px]] || [[File:GGBN mapping three sources.png|Thumb|300px]] |- |'''Pros:''' - only one mapping needed '''Cons:''' - double occurrences at GBIF, since GBIF can't handle associataions ||'''Pros:''' - Only underlying specimens records are provided, no double entries at GBIF '''Cons:''' - two mappings needed (but can be easily copied) ||'''Pros:''' - Only underlying specimens records are provided, no double entries at GBIF - Clear distinction between collections, some managers prefer this mapping type '''Cons:''' - three mappings needed (but can be easily copied) |} ==Mapping without Specimens== Often tissue samples are collected without collecting/killing the whole organism. There are two ways to handle this. {| class="wikitable" ! Mapping eVouchers !! Mapping tissues only |- |style="width: 50%;"|[[File:GGBN mapping evoucher.png|Thumb|300px]] ||[[File:GGBN mapping tissues only.png|Thumb|300px]] |- |In case you want to track an individual, sampled multiple times (e.g. vertebrates caught multiple times and marked individually, trees) you should add an Observation record as top level. This could also be accompanied by images, then called eVoucher. ||In case you don't want to track individuals, just provide your tissue records to GBIF as top level occurrence. |} ==Environmental Samples== Environmental Samples and even more environmental DNA can currently be provided to GGBN and GBIF in a very basic way only. We are working on a proposal how to map these important sample information in a way they deserve it. Have a look at our [[Use_Cases_Enviro | Environmental DNA use case]] =ABCD= ==Metadata of your DNA and tissue bank== Metadata are very important. They will be displayed with every single DNA record in the portal. Be careful with the IPR statements! {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |ContentContact |*Address |Complete Address of the responsible person |Botanic Garden and Botanical Museum Berlin-Dahlem, Freie Universität Berlin, Koenigin-Luise-Str. 6-8, 14195 Berlin, Germany |- |ContentContact |*Email |valid email address; will be used by the web portal for sending you annotations etc. |please use @ instead of masquerades like "[at]" |- |ContentContact |*Name |Person, person team responsible for curation of your DNA Bank |Gabi Droege |- |Description/Representation |*Details |Short text to describe the focus and number of samples in your DNA collection. '''It should include the following phrase:''' ''The DNA bank is part of the Global Genome Biodiversity Network (GGBN) which was founded in 2011. The network provides a technically optimized DNA collection service facility for all biological research accessible via one central web portal. The network promotes deposition of well documented reference DNA samples after project completion or data publication from scientists of other universities and institutions.'' |The DNA bank of the Botanic Garden and Botanical Museum Berlin-Dahlem holds currently a collection of 20000 plant DNA and tissue samples growing constantly. Its core collection focuses on the flora of Germany but it comprises botanical samples collected worldwide. |- |Description/Representation |*@language |language of description |should be "en" as literal |- |Description/Representation |*Title |Short title that describes your DNA collection |DNA Bank of the Herbarium Berolinense |- |IPRStatements/Citation |*Text | |Droege, G. (Ed.) 2008 - (continuously updated): DNA samples of the DNA bank at the BGBM (Botanic Garden and Botanical Museum Berlin-Dahlem). |- |IPRStatements/Citation |*@language |language of citation |should be "en" as literal |- |IPRStatements/Copyright |*Text | |The copyright for any material created by the DNA bank of the BGBM is reserved. The duplication or use of information and data such as texts or images is only permitted with the indication of the source or with prior approval by the BGBM. |- |IPRStatements/Copyright |*@language |language of copyright |should be "en" as literal |- |IPRStatements/TermsOfUse |*Text | |The use of the data is allowed only for non-profit scientific use and for non-profit nature conservation purpose. The data bases or part of it may only be used or copied by the written permission from the legal owner. |- |IPRStatements/TermsOfUse |*@language |language of the terms of use |should be "en" as literal |- | |IconURI |complete url path to the logo of your institution | |- |Owner |*Address |Complete Address of the institution that owns the DNA bank samples and data | |- |Owner/Representation |*Text |name of your institution |Botanic Garden an Botanical Museum Berlin-Dahlem (Freie Universität Berlin) |- |Owner |*URL |path to website of your institution |http://www.bgbm.org |- |RevisionData |*DateModified |date of last modification of your data | |- |TechnicalContact |*Address |Complete Address of the responsible person |Botanic Garden and Botanical Museum Berlin-Dahlem, Freie Universität Berlin, Koenigin-Luise-Str. 6-8, 14195 Berlin, Germany |- |TechnicalContact |*Email |valid email address; will be used by the web portal for sending you annotations etc. |please use @ instead of masquerades like "[at]" |- |TechnicalContact |*Name |Person, person team responsible for technical issues of your database |Gabriele Droege |- |Unit |*RecordBasis |this value must be part of the ABCD vocabulary |must be "MaterialSample" |- |Unit |*KindOfUnit |the type of sample |e.g. DNA |} ==Sample data== ===The sample identifiers=== We use identifier the same way as GBIF does. In addition to the traditional triple ID (see below) you can also provide the GUID. '''Please note that UnitID should be unique in one database!''' Mostly the extraction number is used for it. {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Unit |*<span style="color:red">SourceID</span> |short description of relevant collection |should be "DNA Bank" or "tissue collection" |- |Unit |*<span style="color:red">SourceInstitutionID</span> |short name/abbreviation of relevant institution |BGBM |- |Unit |*<span style="color:red">UnitID</span> |DNA extraction number or tissue number |DNA 123 |- |Unit |UnitGUID |sample record GUID |f6c7fd5a-2f04-4dcb-ba38-0230bf196d30 |} ===GGBN terms=== The following fields are highly recommended, some are mandatory (marked with *), but feel free to map more! <div id="wikinote">'''Please have a look at the [http://terms.tdwg.org/wiki/GGBN_Data_Standard GGBN Data Standard] documentation where all GGBN terms are described in detail. The table below is just listing the most important ones. Since the Nagoya Protocol has come into force we higly recommend to map the Permit fields. They will become mandatory for providing data to GGBN at the end of 2020. Until then all core members must have implemented the required fields in their databases.'''</div> {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- | |*RecordBasis |The specific nature of the data record, controlled vocabulary! |"PreservedSpecimen", "MaterialSample", "FossilSpecimen", "LivingSpecimen", "HumanObservation", "MachineObservation"; for Tissue or DNA samples use "MaterialSample" |- | |*materialSampleType |Classification of kind of physical sample in addition to BasisOfRecord/RecordBasis and Preparation Type |"tissue", "DNA", "specimen" |- |SpecimenUnit/Preparation |*preparationType |description of type of material, free text |or DNA: gDNA, eDNA, aDNA; for tissues/specimens: leaf, muscle, leg, blood |- |SpecimenUnit/Preparation |*preparationDate |date of DNA extraction |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparatioMaterials |extraction kit or protocoll |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparedBy |extraction staff |if unknown type "unknown" |- |SpecimenUnit/Preservation |preservation |preservation of the tissue or DNA |if unknown type "unknown" |- |SpecimenUnit/Loan/Permit |*permitStatus |Information about the presence, absence or other basic status of permits associated with the sample(s), controlled vocabulary! |Permit available, Permit not required, Permit not available, '''Unknown Material collected after 2014-10-12 cannot be in "Unknown" permit status!''' |- |SpecimenUnit/Loan/Permit |*permitStatusQualifier |Description of why a certain permit was not required or why Permit Status is unknown |"no national requirement for a permit at date of access", "officially authorized illegal holder", "collected on private land", "pre-Nagoya" |- |SpecimenUnit/Loan/Permit |*permitType |A permit is a document that allows someone to take an action that otherwise would not be allowed, controlled vocabulary! |Collecting Permit, Import Permit, Export Permit, Intellectual Property Rights, Copyright, Patent, Data use, Phytosanitary, Salvage, Exemption Permit, Material Transfer Agreement, Internationally Recognized Certificate of Compliance, Contract, Memorandum of Understanding, Memorandum of Cooperation, Veterinary Certificate, Human Pathogens, Genetically Modified Organism, Other |- |SpecimenUnit/Loan/Permit |*permitText |The text of a permit related to the gathering/shipping or further details | |- |GGBN/Amplification |amplificationDate |date of amplification; if unknown or general without content you don't have to map it |should be ISO format yyyy-mm-dd |- |GGBN/Amplification |marker | |COX1 |- |GGBN/Amplification |geneticAccessionNumber |the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD; this is a repeatable element, you can provide as much as you want |e.g. AJ45567 |- |GGBN/Amplification |genBankNumber-URI |complete link to the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD | |- |SpecimenUnit |blockedUntil |in case the sample is blocked until a specific date it's nevertheless searchable but customers cannot order it |ISO format |- |SpecimenUnit |blocked |sample is blocked (e.g. because it is consumed), but data are still available |Yes/No |- |GGBN |concentration |concentration of the DNA |65,34 |- |GGBN |@unit |unit of the concentration |µg/ml |- |GGBN |ratioOfAbsorbance260_230 |map only if filled with content |1,2 |- |GGBN |ratioOfAbsorbance260_280 |map only if filled with content |1,8 |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===The voucher identifiers=== {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Associations/UnitAssociation |*UnitID |the UnitID or CatalogueNumber used for GBIF |e.g. the barcode number of your specimens |- |Associations/UnitAssociation |*SourceInstitutionID |the SourceInstitutionID or InstitutionCode used for GBIF |e.g. the acronym of your institution |- |Associations/UnitAssociation |*SourceName |the SourceID or CollectionCode used for GBIF |e.g. the name of the collection where the specimen belongs to, e.g. "Birds" |- |Associations/UnitAssociation |*AssociationType |the Relation between the DNA and the voucher |e.g. "DNA and voucher from same individual" |- |Associations/UnitAssociation |*DatasetAccessPoint |the wrapper url of the voucher record |e.g. "http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar" |} ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} ===Multimedia items of the voucher=== These should be mapped in the specimen mapping, not the DNA mapping. <!-- {| class="wikitable" ! Category ! Element ! Discription ! Comment |- | rowspan="3" | Metadata | Title | | |- | IPR | | |- | TechnicalContact | | |- | Unit | UnitID | | |} --> 2af53affede0d4d951e230759269690684f7be65 986 985 2018-09-27T08:54:04Z JoergHoletschek 3 wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. ==Metadata== Metadata are very important. They will be displayed with every single DNA record in the portal. Be careful with the IPR statements! {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |ContentContact |*Address |Complete Address of the responsible person |Botanic Garden and Botanical Museum Berlin-Dahlem, Freie Universität Berlin, Koenigin-Luise-Str. 6-8, 14195 Berlin, Germany |- |ContentContact |*Email |valid email address; will be used by the web portal for sending you annotations etc. |please use @ instead of masquerades like "[at]" |- |ContentContact |*Name |Person, person team responsible for curation of your DNA Bank |Gabi Droege |- |Description/Representation |*Details |Short text to describe the focus and number of samples in your DNA collection. '''It should include the following phrase:''' ''The DNA bank is part of the Global Genome Biodiversity Network (GGBN) which was founded in 2011. The network provides a technically optimized DNA collection service facility for all biological research accessible via one central web portal. The network promotes deposition of well documented reference DNA samples after project completion or data publication from scientists of other universities and institutions.'' |The DNA bank of the Botanic Garden and Botanical Museum Berlin-Dahlem holds currently a collection of 20000 plant DNA and tissue samples growing constantly. Its core collection focuses on the flora of Germany but it comprises botanical samples collected worldwide. |- |Description/Representation |*@language |language of description |should be "en" as literal |- |Description/Representation |*Title |Short title that describes your DNA collection |DNA Bank of the Herbarium Berolinense |- |IPRStatements/Citation |*Text | |Droege, G. (Ed.) 2008 - (continuously updated): DNA samples of the DNA bank at the BGBM (Botanic Garden and Botanical Museum Berlin-Dahlem). |- |IPRStatements/Citation |*@language |language of citation |should be "en" as literal |- |IPRStatements/Copyright |*Text | |The copyright for any material created by the DNA bank of the BGBM is reserved. The duplication or use of information and data such as texts or images is only permitted with the indication of the source or with prior approval by the BGBM. |- |IPRStatements/Copyright |*@language |language of copyright |should be "en" as literal |- |IPRStatements/TermsOfUse |*Text | |The use of the data is allowed only for non-profit scientific use and for non-profit nature conservation purpose. The data bases or part of it may only be used or copied by the written permission from the legal owner. |- |IPRStatements/TermsOfUse |*@language |language of the terms of use |should be "en" as literal |- | |IconURI |complete url path to the logo of your institution | |- |Owner |*Address |Complete Address of the institution that owns the DNA bank samples and data | |- |Owner/Representation |*Text |name of your institution |Botanic Garden an Botanical Museum Berlin-Dahlem (Freie Universität Berlin) |- |Owner |*URL |path to website of your institution |http://www.bgbm.org |- |RevisionData |*DateModified |date of last modification of your data | |- |TechnicalContact |*Address |Complete Address of the responsible person |Botanic Garden and Botanical Museum Berlin-Dahlem, Freie Universität Berlin, Koenigin-Luise-Str. 6-8, 14195 Berlin, Germany |- |TechnicalContact |*Email |valid email address; will be used by the web portal for sending you annotations etc. |please use @ instead of masquerades like "[at]" |- |TechnicalContact |*Name |Person, person team responsible for technical issues of your database |Gabriele Droege |- |Unit |*RecordBasis |this value must be part of the ABCD vocabulary |must be "MaterialSample" |- |Unit |*KindOfUnit |the type of sample |e.g. DNA |} ==Sample data== ===The sample identifiers=== We use identifier the same way as GBIF does. In addition to the traditional triple ID (see below) you can also provide the GUID. '''Please note that UnitID should be unique in one database!''' Mostly the extraction number is used for it. {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Unit |*<span style="color:red">SourceID</span> |short description of relevant collection |should be "DNA Bank" or "tissue collection" |- |Unit |*<span style="color:red">SourceInstitutionID</span> |short name/abbreviation of relevant institution |BGBM |- |Unit |*<span style="color:red">UnitID</span> |DNA extraction number or tissue number |DNA 123 |- |Unit |UnitGUID |sample record GUID |f6c7fd5a-2f04-4dcb-ba38-0230bf196d30 |} ===GGBN terms=== The following fields are highly recommended, some are mandatory (marked with *), but feel free to map more! <div id="wikinote">'''Please have a look at the [http://terms.tdwg.org/wiki/GGBN_Data_Standard GGBN Data Standard] documentation where all GGBN terms are described in detail. The table below is just listing the most important ones. Since the Nagoya Protocol has come into force we higly recommend to map the Permit fields. They will become mandatory for providing data to GGBN at the end of 2020. Until then all core members must have implemented the required fields in their databases.'''</div> {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- | |*RecordBasis |The specific nature of the data record, controlled vocabulary! |"PreservedSpecimen", "MaterialSample", "FossilSpecimen", "LivingSpecimen", "HumanObservation", "MachineObservation"; for Tissue or DNA samples use "MaterialSample" |- | |*materialSampleType |Classification of kind of physical sample in addition to BasisOfRecord/RecordBasis and Preparation Type |"tissue", "DNA", "specimen" |- |SpecimenUnit/Preparation |*preparationType |description of type of material, free text |or DNA: gDNA, eDNA, aDNA; for tissues/specimens: leaf, muscle, leg, blood |- |SpecimenUnit/Preparation |*preparationDate |date of DNA extraction |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparatioMaterials |extraction kit or protocoll |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparedBy |extraction staff |if unknown type "unknown" |- |SpecimenUnit/Preservation |preservation |preservation of the tissue or DNA |if unknown type "unknown" |- |SpecimenUnit/Loan/Permit |*permitStatus |Information about the presence, absence or other basic status of permits associated with the sample(s), controlled vocabulary! |Permit available, Permit not required, Permit not available, '''Unknown Material collected after 2014-10-12 cannot be in "Unknown" permit status!''' |- |SpecimenUnit/Loan/Permit |*permitStatusQualifier |Description of why a certain permit was not required or why Permit Status is unknown |"no national requirement for a permit at date of access", "officially authorized illegal holder", "collected on private land", "pre-Nagoya" |- |SpecimenUnit/Loan/Permit |*permitType |A permit is a document that allows someone to take an action that otherwise would not be allowed, controlled vocabulary! |Collecting Permit, Import Permit, Export Permit, Intellectual Property Rights, Copyright, Patent, Data use, Phytosanitary, Salvage, Exemption Permit, Material Transfer Agreement, Internationally Recognized Certificate of Compliance, Contract, Memorandum of Understanding, Memorandum of Cooperation, Veterinary Certificate, Human Pathogens, Genetically Modified Organism, Other |- |SpecimenUnit/Loan/Permit |*permitText |The text of a permit related to the gathering/shipping or further details | |- |GGBN/Amplification |amplificationDate |date of amplification; if unknown or general without content you don't have to map it |should be ISO format yyyy-mm-dd |- |GGBN/Amplification |marker | |COX1 |- |GGBN/Amplification |geneticAccessionNumber |the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD; this is a repeatable element, you can provide as much as you want |e.g. AJ45567 |- |GGBN/Amplification |genBankNumber-URI |complete link to the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD | |- |SpecimenUnit |blockedUntil |in case the sample is blocked until a specific date it's nevertheless searchable but customers cannot order it |ISO format |- |SpecimenUnit |blocked |sample is blocked (e.g. because it is consumed), but data are still available |Yes/No |- |GGBN |concentration |concentration of the DNA |65,34 |- |GGBN |@unit |unit of the concentration |µg/ml |- |GGBN |ratioOfAbsorbance260_230 |map only if filled with content |1,2 |- |GGBN |ratioOfAbsorbance260_280 |map only if filled with content |1,8 |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===The voucher identifiers=== {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Associations/UnitAssociation |*UnitID |the UnitID or CatalogueNumber used for GBIF |e.g. the barcode number of your specimens |- |Associations/UnitAssociation |*SourceInstitutionID |the SourceInstitutionID or InstitutionCode used for GBIF |e.g. the acronym of your institution |- |Associations/UnitAssociation |*SourceName |the SourceID or CollectionCode used for GBIF |e.g. the name of the collection where the specimen belongs to, e.g. "Birds" |- |Associations/UnitAssociation |*AssociationType |the Relation between the DNA and the voucher |e.g. "DNA and voucher from same individual" |- |Associations/UnitAssociation |*DatasetAccessPoint |the wrapper url of the voucher record |e.g. "http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar" |} ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} ===Multimedia items of the voucher=== These should be mapped in the specimen mapping, not the DNA mapping. <!-- {| class="wikitable" ! Category ! Element ! Discription ! Comment |- | rowspan="3" | Metadata | Title | | |- | IPR | | |- | TechnicalContact | | |- | Unit | UnitID | | |} --> aed0a9d632cae6d2be94a9af4237e6a5f3ac9898 987 986 2018-09-27T09:11:52Z JoergHoletschek 3 /* Metadata */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |ContentContact |*Address |Complete Address of the responsible person |Botanic Garden and Botanical Museum Berlin-Dahlem, Freie Universität Berlin, Koenigin-Luise-Str. 6-8, 14195 Berlin, Germany |- |ContentContact |*Email |valid email address; will be used by the web portal for sending you annotations etc. |please use @ instead of masquerades like "[at]" |- |ContentContact |*Name |Person, person team responsible for curation of your DNA Bank |Gabi Droege |- |Description/Representation |*Details |Short text to describe the focus and number of samples in your DNA collection. '''It should include the following phrase:''' ''The DNA bank is part of the Global Genome Biodiversity Network (GGBN) which was founded in 2011. The network provides a technically optimized DNA collection service facility for all biological research accessible via one central web portal. The network promotes deposition of well documented reference DNA samples after project completion or data publication from scientists of other universities and institutions.'' |The DNA bank of the Botanic Garden and Botanical Museum Berlin-Dahlem holds currently a collection of 20000 plant DNA and tissue samples growing constantly. Its core collection focuses on the flora of Germany but it comprises botanical samples collected worldwide. |- |Description/Representation |*@language |language of description |should be "en" as literal |- |Description/Representation |*Title |Short title that describes your DNA collection |DNA Bank of the Herbarium Berolinense |- |IPRStatements/Citation |*Text | |Droege, G. (Ed.) 2008 - (continuously updated): DNA samples of the DNA bank at the BGBM (Botanic Garden and Botanical Museum Berlin-Dahlem). |- |IPRStatements/Citation |*@language |language of citation |should be "en" as literal |- |IPRStatements/Copyright |*Text | |The copyright for any material created by the DNA bank of the BGBM is reserved. The duplication or use of information and data such as texts or images is only permitted with the indication of the source or with prior approval by the BGBM. |- |IPRStatements/Copyright |*@language |language of copyright |should be "en" as literal |- |IPRStatements/TermsOfUse |*Text | |The use of the data is allowed only for non-profit scientific use and for non-profit nature conservation purpose. The data bases or part of it may only be used or copied by the written permission from the legal owner. |- |IPRStatements/TermsOfUse |*@language |language of the terms of use |should be "en" as literal |- | |IconURI |complete url path to the logo of your institution | |- |Owner |*Address |Complete Address of the institution that owns the DNA bank samples and data | |- |Owner/Representation |*Text |name of your institution |Botanic Garden an Botanical Museum Berlin-Dahlem (Freie Universität Berlin) |- |Owner |*URL |path to website of your institution |http://www.bgbm.org |- |RevisionData |*DateModified |date of last modification of your data | |- |TechnicalContact |*Address |Complete Address of the responsible person |Botanic Garden and Botanical Museum Berlin-Dahlem, Freie Universität Berlin, Koenigin-Luise-Str. 6-8, 14195 Berlin, Germany |- |TechnicalContact |*Email |valid email address; will be used by the web portal for sending you annotations etc. |please use @ instead of masquerades like "[at]" |- |TechnicalContact |*Name |Person, person team responsible for technical issues of your database |Gabriele Droege |- |Unit |*RecordBasis |this value must be part of the ABCD vocabulary |must be "MaterialSample" |- |Unit |*KindOfUnit |the type of sample |e.g. DNA |} ==Sample data== ===The sample identifiers=== We use identifier the same way as GBIF does. In addition to the traditional triple ID (see below) you can also provide the GUID. '''Please note that UnitID should be unique in one database!''' Mostly the extraction number is used for it. {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Unit |*<span style="color:red">SourceID</span> |short description of relevant collection |should be "DNA Bank" or "tissue collection" |- |Unit |*<span style="color:red">SourceInstitutionID</span> |short name/abbreviation of relevant institution |BGBM |- |Unit |*<span style="color:red">UnitID</span> |DNA extraction number or tissue number |DNA 123 |- |Unit |UnitGUID |sample record GUID |f6c7fd5a-2f04-4dcb-ba38-0230bf196d30 |} ===GGBN terms=== The following fields are highly recommended, some are mandatory (marked with *), but feel free to map more! <div id="wikinote">'''Please have a look at the [http://terms.tdwg.org/wiki/GGBN_Data_Standard GGBN Data Standard] documentation where all GGBN terms are described in detail. The table below is just listing the most important ones. Since the Nagoya Protocol has come into force we higly recommend to map the Permit fields. They will become mandatory for providing data to GGBN at the end of 2020. Until then all core members must have implemented the required fields in their databases.'''</div> {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- | |*RecordBasis |The specific nature of the data record, controlled vocabulary! |"PreservedSpecimen", "MaterialSample", "FossilSpecimen", "LivingSpecimen", "HumanObservation", "MachineObservation"; for Tissue or DNA samples use "MaterialSample" |- | |*materialSampleType |Classification of kind of physical sample in addition to BasisOfRecord/RecordBasis and Preparation Type |"tissue", "DNA", "specimen" |- |SpecimenUnit/Preparation |*preparationType |description of type of material, free text |or DNA: gDNA, eDNA, aDNA; for tissues/specimens: leaf, muscle, leg, blood |- |SpecimenUnit/Preparation |*preparationDate |date of DNA extraction |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparatioMaterials |extraction kit or protocoll |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparedBy |extraction staff |if unknown type "unknown" |- |SpecimenUnit/Preservation |preservation |preservation of the tissue or DNA |if unknown type "unknown" |- |SpecimenUnit/Loan/Permit |*permitStatus |Information about the presence, absence or other basic status of permits associated with the sample(s), controlled vocabulary! |Permit available, Permit not required, Permit not available, '''Unknown Material collected after 2014-10-12 cannot be in "Unknown" permit status!''' |- |SpecimenUnit/Loan/Permit |*permitStatusQualifier |Description of why a certain permit was not required or why Permit Status is unknown |"no national requirement for a permit at date of access", "officially authorized illegal holder", "collected on private land", "pre-Nagoya" |- |SpecimenUnit/Loan/Permit |*permitType |A permit is a document that allows someone to take an action that otherwise would not be allowed, controlled vocabulary! |Collecting Permit, Import Permit, Export Permit, Intellectual Property Rights, Copyright, Patent, Data use, Phytosanitary, Salvage, Exemption Permit, Material Transfer Agreement, Internationally Recognized Certificate of Compliance, Contract, Memorandum of Understanding, Memorandum of Cooperation, Veterinary Certificate, Human Pathogens, Genetically Modified Organism, Other |- |SpecimenUnit/Loan/Permit |*permitText |The text of a permit related to the gathering/shipping or further details | |- |GGBN/Amplification |amplificationDate |date of amplification; if unknown or general without content you don't have to map it |should be ISO format yyyy-mm-dd |- |GGBN/Amplification |marker | |COX1 |- |GGBN/Amplification |geneticAccessionNumber |the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD; this is a repeatable element, you can provide as much as you want |e.g. AJ45567 |- |GGBN/Amplification |genBankNumber-URI |complete link to the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD | |- |SpecimenUnit |blockedUntil |in case the sample is blocked until a specific date it's nevertheless searchable but customers cannot order it |ISO format |- |SpecimenUnit |blocked |sample is blocked (e.g. because it is consumed), but data are still available |Yes/No |- |GGBN |concentration |concentration of the DNA |65,34 |- |GGBN |@unit |unit of the concentration |µg/ml |- |GGBN |ratioOfAbsorbance260_230 |map only if filled with content |1,2 |- |GGBN |ratioOfAbsorbance260_280 |map only if filled with content |1,8 |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===The voucher identifiers=== {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Associations/UnitAssociation |*UnitID |the UnitID or CatalogueNumber used for GBIF |e.g. the barcode number of your specimens |- |Associations/UnitAssociation |*SourceInstitutionID |the SourceInstitutionID or InstitutionCode used for GBIF |e.g. the acronym of your institution |- |Associations/UnitAssociation |*SourceName |the SourceID or CollectionCode used for GBIF |e.g. the name of the collection where the specimen belongs to, e.g. "Birds" |- |Associations/UnitAssociation |*AssociationType |the Relation between the DNA and the voucher |e.g. "DNA and voucher from same individual" |- |Associations/UnitAssociation |*DatasetAccessPoint |the wrapper url of the voucher record |e.g. "http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar" |} ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} ===Multimedia items of the voucher=== These should be mapped in the specimen mapping, not the DNA mapping. <!-- {| class="wikitable" ! Category ! Element ! Discription ! Comment |- | rowspan="3" | Metadata | Title | | |- | IPR | | |- | TechnicalContact | | |- | Unit | UnitID | | |} --> 3f59756b4f00f2e1ea829a763bf3ae891d10640f 988 987 2018-09-27T09:13:44Z JoergHoletschek 3 wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |ContentContact |*Address |Complete Address of the responsible person |Botanic Garden and Botanical Museum Berlin-Dahlem, Freie Universität Berlin, Koenigin-Luise-Str. 6-8, 14195 Berlin, Germany |- |ContentContact |*Email |valid email address; will be used by the web portal for sending you annotations etc. |please use @ instead of masquerades like "[at]" |- |ContentContact |*Name |Person, person team responsible for curation of your DNA Bank |Gabi Droege |- |Description/Representation |*Details |Short text to describe the focus and number of samples in your DNA collection. '''It should include the following phrase:''' ''The DNA bank is part of the Global Genome Biodiversity Network (GGBN) which was founded in 2011. The network provides a technically optimized DNA collection service facility for all biological research accessible via one central web portal. The network promotes deposition of well documented reference DNA samples after project completion or data publication from scientists of other universities and institutions.'' |The DNA bank of the Botanic Garden and Botanical Museum Berlin-Dahlem holds currently a collection of 20000 plant DNA and tissue samples growing constantly. Its core collection focuses on the flora of Germany but it comprises botanical samples collected worldwide. |- |Description/Representation |*@language |language of description |should be "en" as literal |- |Description/Representation |*Title |Short title that describes your DNA collection |DNA Bank of the Herbarium Berolinense |- |IPRStatements/Citation |*Text | |Droege, G. (Ed.) 2008 - (continuously updated): DNA samples of the DNA bank at the BGBM (Botanic Garden and Botanical Museum Berlin-Dahlem). |- |IPRStatements/Citation |*@language |language of citation |should be "en" as literal |- |IPRStatements/Copyright |*Text | |The copyright for any material created by the DNA bank of the BGBM is reserved. The duplication or use of information and data such as texts or images is only permitted with the indication of the source or with prior approval by the BGBM. |- |IPRStatements/Copyright |*@language |language of copyright |should be "en" as literal |- |IPRStatements/TermsOfUse |*Text | |The use of the data is allowed only for non-profit scientific use and for non-profit nature conservation purpose. The data bases or part of it may only be used or copied by the written permission from the legal owner. |- |IPRStatements/TermsOfUse |*@language |language of the terms of use |should be "en" as literal |- | |IconURI |complete url path to the logo of your institution | |- |Owner |*Address |Complete Address of the institution that owns the DNA bank samples and data | |- |Owner/Representation |*Text |name of your institution |Botanic Garden an Botanical Museum Berlin-Dahlem (Freie Universität Berlin) |- |Owner |*URL |path to website of your institution |http://www.bgbm.org |- |RevisionData |*DateModified |date of last modification of your data | |- |TechnicalContact |*Address |Complete Address of the responsible person |Botanic Garden and Botanical Museum Berlin-Dahlem, Freie Universität Berlin, Koenigin-Luise-Str. 6-8, 14195 Berlin, Germany |- |TechnicalContact |*Email |valid email address; will be used by the web portal for sending you annotations etc. |please use @ instead of masquerades like "[at]" |- |TechnicalContact |*Name |Person, person team responsible for technical issues of your database |Gabriele Droege |- |Unit |*RecordBasis |this value must be part of the ABCD vocabulary |must be "MaterialSample" |- |Unit |*KindOfUnit |the type of sample |e.g. DNA |} ==Sample data== ===The sample identifiers=== We use identifier the same way as GBIF does. In addition to the traditional triple ID (see below) you can also provide the GUID. '''Please note that UnitID should be unique in one database!''' Mostly the extraction number is used for it. {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Unit |*<span style="color:red">SourceID</span> |short description of relevant collection |should be "DNA Bank" or "tissue collection" |- |Unit |*<span style="color:red">SourceInstitutionID</span> |short name/abbreviation of relevant institution |BGBM |- |Unit |*<span style="color:red">UnitID</span> |DNA extraction number or tissue number |DNA 123 |- |Unit |UnitGUID |sample record GUID |f6c7fd5a-2f04-4dcb-ba38-0230bf196d30 |} ===GGBN terms=== The following fields are highly recommended, some are mandatory (marked with *), but feel free to map more! <div id="wikinote">'''Please have a look at the [http://terms.tdwg.org/wiki/GGBN_Data_Standard GGBN Data Standard] documentation where all GGBN terms are described in detail. The table below is just listing the most important ones. Since the Nagoya Protocol has come into force we higly recommend to map the Permit fields. They will become mandatory for providing data to GGBN at the end of 2020. Until then all core members must have implemented the required fields in their databases.'''</div> {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- | |*RecordBasis |The specific nature of the data record, controlled vocabulary! |"PreservedSpecimen", "MaterialSample", "FossilSpecimen", "LivingSpecimen", "HumanObservation", "MachineObservation"; for Tissue or DNA samples use "MaterialSample" |- | |*materialSampleType |Classification of kind of physical sample in addition to BasisOfRecord/RecordBasis and Preparation Type |"tissue", "DNA", "specimen" |- |SpecimenUnit/Preparation |*preparationType |description of type of material, free text |or DNA: gDNA, eDNA, aDNA; for tissues/specimens: leaf, muscle, leg, blood |- |SpecimenUnit/Preparation |*preparationDate |date of DNA extraction |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparatioMaterials |extraction kit or protocoll |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparedBy |extraction staff |if unknown type "unknown" |- |SpecimenUnit/Preservation |preservation |preservation of the tissue or DNA |if unknown type "unknown" |- |SpecimenUnit/Loan/Permit |*permitStatus |Information about the presence, absence or other basic status of permits associated with the sample(s), controlled vocabulary! |Permit available, Permit not required, Permit not available, '''Unknown Material collected after 2014-10-12 cannot be in "Unknown" permit status!''' |- |SpecimenUnit/Loan/Permit |*permitStatusQualifier |Description of why a certain permit was not required or why Permit Status is unknown |"no national requirement for a permit at date of access", "officially authorized illegal holder", "collected on private land", "pre-Nagoya" |- |SpecimenUnit/Loan/Permit |*permitType |A permit is a document that allows someone to take an action that otherwise would not be allowed, controlled vocabulary! |Collecting Permit, Import Permit, Export Permit, Intellectual Property Rights, Copyright, Patent, Data use, Phytosanitary, Salvage, Exemption Permit, Material Transfer Agreement, Internationally Recognized Certificate of Compliance, Contract, Memorandum of Understanding, Memorandum of Cooperation, Veterinary Certificate, Human Pathogens, Genetically Modified Organism, Other |- |SpecimenUnit/Loan/Permit |*permitText |The text of a permit related to the gathering/shipping or further details | |- |GGBN/Amplification |amplificationDate |date of amplification; if unknown or general without content you don't have to map it |should be ISO format yyyy-mm-dd |- |GGBN/Amplification |marker | |COX1 |- |GGBN/Amplification |geneticAccessionNumber |the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD; this is a repeatable element, you can provide as much as you want |e.g. AJ45567 |- |GGBN/Amplification |genBankNumber-URI |complete link to the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD | |- |SpecimenUnit |blockedUntil |in case the sample is blocked until a specific date it's nevertheless searchable but customers cannot order it |ISO format |- |SpecimenUnit |blocked |sample is blocked (e.g. because it is consumed), but data are still available |Yes/No |- |GGBN |concentration |concentration of the DNA |65,34 |- |GGBN |@unit |unit of the concentration |µg/ml |- |GGBN |ratioOfAbsorbance260_230 |map only if filled with content |1,2 |- |GGBN |ratioOfAbsorbance260_280 |map only if filled with content |1,8 |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===The voucher identifiers=== {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Associations/UnitAssociation |*UnitID |the UnitID or CatalogueNumber used for GBIF |e.g. the barcode number of your specimens |- |Associations/UnitAssociation |*SourceInstitutionID |the SourceInstitutionID or InstitutionCode used for GBIF |e.g. the acronym of your institution |- |Associations/UnitAssociation |*SourceName |the SourceID or CollectionCode used for GBIF |e.g. the name of the collection where the specimen belongs to, e.g. "Birds" |- |Associations/UnitAssociation |*AssociationType |the Relation between the DNA and the voucher |e.g. "DNA and voucher from same individual" |- |Associations/UnitAssociation |*DatasetAccessPoint |the wrapper url of the voucher record |e.g. "http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar" |} ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} ===Multimedia items of the voucher=== These should be mapped in the specimen mapping, not the DNA mapping. <!-- {| class="wikitable" ! Category ! Element ! Discription ! Comment |- | rowspan="3" | Metadata | Title | | |- | IPR | | |- | TechnicalContact | | |- | Unit | UnitID | | |} --> beb98bdd7a3293d3a71a11b275e11e12d6c96823 989 988 2018-09-27T09:19:50Z JoergHoletschek 3 wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |rowspan="4" | ContentContact |Name * |Person, person team responsible for curation of your DNA Bank |Gabi Droege |- |Address |Complete Address of the responsible person |Botanic Garden and Botanical Museum Berlin-Dahlem, Freie Universität Berlin, Koenigin-Luise-Str. 6-8, 14195 Berlin, Germany |- |Email |valid email address; will be used by the web portal for sending you annotations etc. |please use @ instead of masquerades like "[at]" |- |Phone |valid email address; will be used by the web portal for sending you annotations etc. |please use @ instead of masquerades like "[at]" |- |Description/Representation |*Details |Short text to describe the focus and number of samples in your DNA collection. '''It should include the following phrase:''' ''The DNA bank is part of the Global Genome Biodiversity Network (GGBN) which was founded in 2011. The network provides a technically optimized DNA collection service facility for all biological research accessible via one central web portal. The network promotes deposition of well documented reference DNA samples after project completion or data publication from scientists of other universities and institutions.'' |The DNA bank of the Botanic Garden and Botanical Museum Berlin-Dahlem holds currently a collection of 20000 plant DNA and tissue samples growing constantly. Its core collection focuses on the flora of Germany but it comprises botanical samples collected worldwide. |- |Description/Representation |*@language |language of description |should be "en" as literal |- |Description/Representation |*Title |Short title that describes your DNA collection |DNA Bank of the Herbarium Berolinense |- |IPRStatements/Citation |*Text | |Droege, G. (Ed.) 2008 - (continuously updated): DNA samples of the DNA bank at the BGBM (Botanic Garden and Botanical Museum Berlin-Dahlem). |- |IPRStatements/Citation |*@language |language of citation |should be "en" as literal |- |IPRStatements/Copyright |*Text | |The copyright for any material created by the DNA bank of the BGBM is reserved. The duplication or use of information and data such as texts or images is only permitted with the indication of the source or with prior approval by the BGBM. |- |IPRStatements/Copyright |*@language |language of copyright |should be "en" as literal |- |IPRStatements/TermsOfUse |*Text | |The use of the data is allowed only for non-profit scientific use and for non-profit nature conservation purpose. The data bases or part of it may only be used or copied by the written permission from the legal owner. |- |IPRStatements/TermsOfUse |*@language |language of the terms of use |should be "en" as literal |- | |IconURI |complete url path to the logo of your institution | |- |Owner |*Address |Complete Address of the institution that owns the DNA bank samples and data | |- |Owner/Representation |*Text |name of your institution |Botanic Garden an Botanical Museum Berlin-Dahlem (Freie Universität Berlin) |- |Owner |*URL |path to website of your institution |http://www.bgbm.org |- |RevisionData |*DateModified |date of last modification of your data | |- |TechnicalContact |*Address |Complete Address of the responsible person |Botanic Garden and Botanical Museum Berlin-Dahlem, Freie Universität Berlin, Koenigin-Luise-Str. 6-8, 14195 Berlin, Germany |- |TechnicalContact |*Email |valid email address; will be used by the web portal for sending you annotations etc. |please use @ instead of masquerades like "[at]" |- |TechnicalContact |*Name |Person, person team responsible for technical issues of your database |Gabriele Droege |- |Unit |*RecordBasis |this value must be part of the ABCD vocabulary |must be "MaterialSample" |- |Unit |*KindOfUnit |the type of sample |e.g. DNA |} ==Sample data== ===The sample identifiers=== We use identifier the same way as GBIF does. In addition to the traditional triple ID (see below) you can also provide the GUID. '''Please note that UnitID should be unique in one database!''' Mostly the extraction number is used for it. {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Unit |*<span style="color:red">SourceID</span> |short description of relevant collection |should be "DNA Bank" or "tissue collection" |- |Unit |*<span style="color:red">SourceInstitutionID</span> |short name/abbreviation of relevant institution |BGBM |- |Unit |*<span style="color:red">UnitID</span> |DNA extraction number or tissue number |DNA 123 |- |Unit |UnitGUID |sample record GUID |f6c7fd5a-2f04-4dcb-ba38-0230bf196d30 |} ===GGBN terms=== The following fields are highly recommended, some are mandatory (marked with *), but feel free to map more! <div id="wikinote">'''Please have a look at the [http://terms.tdwg.org/wiki/GGBN_Data_Standard GGBN Data Standard] documentation where all GGBN terms are described in detail. The table below is just listing the most important ones. Since the Nagoya Protocol has come into force we higly recommend to map the Permit fields. They will become mandatory for providing data to GGBN at the end of 2020. Until then all core members must have implemented the required fields in their databases.'''</div> {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- | |*RecordBasis |The specific nature of the data record, controlled vocabulary! |"PreservedSpecimen", "MaterialSample", "FossilSpecimen", "LivingSpecimen", "HumanObservation", "MachineObservation"; for Tissue or DNA samples use "MaterialSample" |- | |*materialSampleType |Classification of kind of physical sample in addition to BasisOfRecord/RecordBasis and Preparation Type |"tissue", "DNA", "specimen" |- |SpecimenUnit/Preparation |*preparationType |description of type of material, free text |or DNA: gDNA, eDNA, aDNA; for tissues/specimens: leaf, muscle, leg, blood |- |SpecimenUnit/Preparation |*preparationDate |date of DNA extraction |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparatioMaterials |extraction kit or protocoll |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparedBy |extraction staff |if unknown type "unknown" |- |SpecimenUnit/Preservation |preservation |preservation of the tissue or DNA |if unknown type "unknown" |- |SpecimenUnit/Loan/Permit |*permitStatus |Information about the presence, absence or other basic status of permits associated with the sample(s), controlled vocabulary! |Permit available, Permit not required, Permit not available, '''Unknown Material collected after 2014-10-12 cannot be in "Unknown" permit status!''' |- |SpecimenUnit/Loan/Permit |*permitStatusQualifier |Description of why a certain permit was not required or why Permit Status is unknown |"no national requirement for a permit at date of access", "officially authorized illegal holder", "collected on private land", "pre-Nagoya" |- |SpecimenUnit/Loan/Permit |*permitType |A permit is a document that allows someone to take an action that otherwise would not be allowed, controlled vocabulary! |Collecting Permit, Import Permit, Export Permit, Intellectual Property Rights, Copyright, Patent, Data use, Phytosanitary, Salvage, Exemption Permit, Material Transfer Agreement, Internationally Recognized Certificate of Compliance, Contract, Memorandum of Understanding, Memorandum of Cooperation, Veterinary Certificate, Human Pathogens, Genetically Modified Organism, Other |- |SpecimenUnit/Loan/Permit |*permitText |The text of a permit related to the gathering/shipping or further details | |- |GGBN/Amplification |amplificationDate |date of amplification; if unknown or general without content you don't have to map it |should be ISO format yyyy-mm-dd |- |GGBN/Amplification |marker | |COX1 |- |GGBN/Amplification |geneticAccessionNumber |the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD; this is a repeatable element, you can provide as much as you want |e.g. AJ45567 |- |GGBN/Amplification |genBankNumber-URI |complete link to the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD | |- |SpecimenUnit |blockedUntil |in case the sample is blocked until a specific date it's nevertheless searchable but customers cannot order it |ISO format |- |SpecimenUnit |blocked |sample is blocked (e.g. because it is consumed), but data are still available |Yes/No |- |GGBN |concentration |concentration of the DNA |65,34 |- |GGBN |@unit |unit of the concentration |µg/ml |- |GGBN |ratioOfAbsorbance260_230 |map only if filled with content |1,2 |- |GGBN |ratioOfAbsorbance260_280 |map only if filled with content |1,8 |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===The voucher identifiers=== {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Associations/UnitAssociation |*UnitID |the UnitID or CatalogueNumber used for GBIF |e.g. the barcode number of your specimens |- |Associations/UnitAssociation |*SourceInstitutionID |the SourceInstitutionID or InstitutionCode used for GBIF |e.g. the acronym of your institution |- |Associations/UnitAssociation |*SourceName |the SourceID or CollectionCode used for GBIF |e.g. the name of the collection where the specimen belongs to, e.g. "Birds" |- |Associations/UnitAssociation |*AssociationType |the Relation between the DNA and the voucher |e.g. "DNA and voucher from same individual" |- |Associations/UnitAssociation |*DatasetAccessPoint |the wrapper url of the voucher record |e.g. "http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar" |} ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} ===Multimedia items of the voucher=== These should be mapped in the specimen mapping, not the DNA mapping. {| class="wikitable" ! Category ! Element ! Discription ! Comment |- | rowspan="3" | Metadata | Title | | |- | IPR | | |- | TechnicalContact | | |- | Unit | UnitID | | |} 454b697c43d823e23a9d486039e54bc1e54fbea9 990 989 2018-09-27T09:34:03Z JoergHoletschek 3 /* Metadata */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |rowspan="4" | ContentContact |Name* |rowspan="4" | Contact information for the curator of the dataset. |John Doe |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone |john.doe@acme.com |- |rowspan="4" | TechnicalContact |Name* |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |John Doe |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone |john.doe@acme.com |- |rowspan="2" | RevisionData |DateCreated |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |2018-07-19 |- |DateModified* |2018-09-27 |- |Description/Representation |*Details |Short text to describe the focus and number of samples in your DNA collection. '''It should include the following phrase:''' ''The DNA bank is part of the Global Genome Biodiversity Network (GGBN) which was founded in 2011. The network provides a technically optimized DNA collection service facility for all biological research accessible via one central web portal. The network promotes deposition of well documented reference DNA samples after project completion or data publication from scientists of other universities and institutions.'' |The DNA bank of the Botanic Garden and Botanical Museum Berlin-Dahlem holds currently a collection of 20000 plant DNA and tissue samples growing constantly. Its core collection focuses on the flora of Germany but it comprises botanical samples collected worldwide. |- |Description/Representation |*@language |language of description |should be "en" as literal |- |Description/Representation |*Title |Short title that describes your DNA collection |DNA Bank of the Herbarium Berolinense |- |IPRStatements/Citation |*Text | |Droege, G. (Ed.) 2008 - (continuously updated): DNA samples of the DNA bank at the BGBM (Botanic Garden and Botanical Museum Berlin-Dahlem). |- |IPRStatements/Citation |*@language |language of citation |should be "en" as literal |- |IPRStatements/Copyright |*Text | |The copyright for any material created by the DNA bank of the BGBM is reserved. The duplication or use of information and data such as texts or images is only permitted with the indication of the source or with prior approval by the BGBM. |- |IPRStatements/Copyright |*@language |language of copyright |should be "en" as literal |- |IPRStatements/TermsOfUse |*Text | |The use of the data is allowed only for non-profit scientific use and for non-profit nature conservation purpose. The data bases or part of it may only be used or copied by the written permission from the legal owner. |- |IPRStatements/TermsOfUse |*@language |language of the terms of use |should be "en" as literal |- | |IconURI |complete url path to the logo of your institution | |- |Owner |*Address |Complete Address of the institution that owns the DNA bank samples and data | |- |Owner/Representation |*Text |name of your institution |Botanic Garden an Botanical Museum Berlin-Dahlem (Freie Universität Berlin) |- |Owner |*URL |path to website of your institution |http://www.bgbm.org |- |RevisionData |*DateModified |date of last modification of your data | |- |TechnicalContact |*Address |Complete Address of the responsible person |Botanic Garden and Botanical Museum Berlin-Dahlem, Freie Universität Berlin, Koenigin-Luise-Str. 6-8, 14195 Berlin, Germany |- |TechnicalContact |*Email |valid email address; will be used by the web portal for sending you annotations etc. |please use @ instead of masquerades like "[at]" |- |TechnicalContact |*Name |Person, person team responsible for technical issues of your database |Gabriele Droege |- |Unit |*RecordBasis |this value must be part of the ABCD vocabulary |must be "MaterialSample" |- |Unit |*KindOfUnit |the type of sample |e.g. DNA |} ==Sample data== ===The sample identifiers=== We use identifier the same way as GBIF does. In addition to the traditional triple ID (see below) you can also provide the GUID. '''Please note that UnitID should be unique in one database!''' Mostly the extraction number is used for it. {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Unit |*<span style="color:red">SourceID</span> |short description of relevant collection |should be "DNA Bank" or "tissue collection" |- |Unit |*<span style="color:red">SourceInstitutionID</span> |short name/abbreviation of relevant institution |BGBM |- |Unit |*<span style="color:red">UnitID</span> |DNA extraction number or tissue number |DNA 123 |- |Unit |UnitGUID |sample record GUID |f6c7fd5a-2f04-4dcb-ba38-0230bf196d30 |} ===GGBN terms=== The following fields are highly recommended, some are mandatory (marked with *), but feel free to map more! <div id="wikinote">'''Please have a look at the [http://terms.tdwg.org/wiki/GGBN_Data_Standard GGBN Data Standard] documentation where all GGBN terms are described in detail. The table below is just listing the most important ones. Since the Nagoya Protocol has come into force we higly recommend to map the Permit fields. They will become mandatory for providing data to GGBN at the end of 2020. Until then all core members must have implemented the required fields in their databases.'''</div> {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- | |*RecordBasis |The specific nature of the data record, controlled vocabulary! |"PreservedSpecimen", "MaterialSample", "FossilSpecimen", "LivingSpecimen", "HumanObservation", "MachineObservation"; for Tissue or DNA samples use "MaterialSample" |- | |*materialSampleType |Classification of kind of physical sample in addition to BasisOfRecord/RecordBasis and Preparation Type |"tissue", "DNA", "specimen" |- |SpecimenUnit/Preparation |*preparationType |description of type of material, free text |or DNA: gDNA, eDNA, aDNA; for tissues/specimens: leaf, muscle, leg, blood |- |SpecimenUnit/Preparation |*preparationDate |date of DNA extraction |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparatioMaterials |extraction kit or protocoll |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparedBy |extraction staff |if unknown type "unknown" |- |SpecimenUnit/Preservation |preservation |preservation of the tissue or DNA |if unknown type "unknown" |- |SpecimenUnit/Loan/Permit |*permitStatus |Information about the presence, absence or other basic status of permits associated with the sample(s), controlled vocabulary! |Permit available, Permit not required, Permit not available, '''Unknown Material collected after 2014-10-12 cannot be in "Unknown" permit status!''' |- |SpecimenUnit/Loan/Permit |*permitStatusQualifier |Description of why a certain permit was not required or why Permit Status is unknown |"no national requirement for a permit at date of access", "officially authorized illegal holder", "collected on private land", "pre-Nagoya" |- |SpecimenUnit/Loan/Permit |*permitType |A permit is a document that allows someone to take an action that otherwise would not be allowed, controlled vocabulary! |Collecting Permit, Import Permit, Export Permit, Intellectual Property Rights, Copyright, Patent, Data use, Phytosanitary, Salvage, Exemption Permit, Material Transfer Agreement, Internationally Recognized Certificate of Compliance, Contract, Memorandum of Understanding, Memorandum of Cooperation, Veterinary Certificate, Human Pathogens, Genetically Modified Organism, Other |- |SpecimenUnit/Loan/Permit |*permitText |The text of a permit related to the gathering/shipping or further details | |- |GGBN/Amplification |amplificationDate |date of amplification; if unknown or general without content you don't have to map it |should be ISO format yyyy-mm-dd |- |GGBN/Amplification |marker | |COX1 |- |GGBN/Amplification |geneticAccessionNumber |the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD; this is a repeatable element, you can provide as much as you want |e.g. AJ45567 |- |GGBN/Amplification |genBankNumber-URI |complete link to the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD | |- |SpecimenUnit |blockedUntil |in case the sample is blocked until a specific date it's nevertheless searchable but customers cannot order it |ISO format |- |SpecimenUnit |blocked |sample is blocked (e.g. because it is consumed), but data are still available |Yes/No |- |GGBN |concentration |concentration of the DNA |65,34 |- |GGBN |@unit |unit of the concentration |µg/ml |- |GGBN |ratioOfAbsorbance260_230 |map only if filled with content |1,2 |- |GGBN |ratioOfAbsorbance260_280 |map only if filled with content |1,8 |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===The voucher identifiers=== {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Associations/UnitAssociation |*UnitID |the UnitID or CatalogueNumber used for GBIF |e.g. the barcode number of your specimens |- |Associations/UnitAssociation |*SourceInstitutionID |the SourceInstitutionID or InstitutionCode used for GBIF |e.g. the acronym of your institution |- |Associations/UnitAssociation |*SourceName |the SourceID or CollectionCode used for GBIF |e.g. the name of the collection where the specimen belongs to, e.g. "Birds" |- |Associations/UnitAssociation |*AssociationType |the Relation between the DNA and the voucher |e.g. "DNA and voucher from same individual" |- |Associations/UnitAssociation |*DatasetAccessPoint |the wrapper url of the voucher record |e.g. "http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar" |} ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} ===Multimedia items of the voucher=== These should be mapped in the specimen mapping, not the DNA mapping. {| class="wikitable" ! Category ! Element ! Discription ! Comment |- | rowspan="3" | Metadata | Title | | |- | IPR | | |- | TechnicalContact | | |- | Unit | UnitID | | |} 6cbed1f383ba8547b4d102f4e3eeb41b05546d16 991 990 2018-09-27T09:38:16Z JoergHoletschek 3 /* Metadata */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |rowspan="4" | ContentContact |Name* |rowspan="4" | Contact information for the curator of the dataset. |John Doe |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone |john.doe@acme.com |- |rowspan="4" | TechnicalContact |Name* |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |John Doe |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone |john.doe@acme.com |- |rowspan="2" | RevisionData |DateCreated |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |2018-07-19 |- |DateModified* |2018-09-27 |- |rowspan="3" | Version |DateIssued |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |2018-09-27 |- |Major* |3 |- |Minor |1 |- |Description/Representation |*Details |Short text to describe the focus and number of samples in your DNA collection. '''It should include the following phrase:''' ''The DNA bank is part of the Global Genome Biodiversity Network (GGBN) which was founded in 2011. The network provides a technically optimized DNA collection service facility for all biological research accessible via one central web portal. The network promotes deposition of well documented reference DNA samples after project completion or data publication from scientists of other universities and institutions.'' |The DNA bank of the Botanic Garden and Botanical Museum Berlin-Dahlem holds currently a collection of 20000 plant DNA and tissue samples growing constantly. Its core collection focuses on the flora of Germany but it comprises botanical samples collected worldwide. |- |Description/Representation |*@language |language of description |should be "en" as literal |- |Description/Representation |*Title |Short title that describes your DNA collection |DNA Bank of the Herbarium Berolinense |- |IPRStatements/Citation |*Text | |Droege, G. (Ed.) 2008 - (continuously updated): DNA samples of the DNA bank at the BGBM (Botanic Garden and Botanical Museum Berlin-Dahlem). |- |IPRStatements/Citation |*@language |language of citation |should be "en" as literal |- |IPRStatements/Copyright |*Text | |The copyright for any material created by the DNA bank of the BGBM is reserved. The duplication or use of information and data such as texts or images is only permitted with the indication of the source or with prior approval by the BGBM. |- |IPRStatements/Copyright |*@language |language of copyright |should be "en" as literal |- |IPRStatements/TermsOfUse |*Text | |The use of the data is allowed only for non-profit scientific use and for non-profit nature conservation purpose. The data bases or part of it may only be used or copied by the written permission from the legal owner. |- |IPRStatements/TermsOfUse |*@language |language of the terms of use |should be "en" as literal |- | |IconURI |complete url path to the logo of your institution | |- |Owner |*Address |Complete Address of the institution that owns the DNA bank samples and data | |- |Owner/Representation |*Text |name of your institution |Botanic Garden an Botanical Museum Berlin-Dahlem (Freie Universität Berlin) |- |Owner |*URL |path to website of your institution |http://www.bgbm.org |- |Unit |*RecordBasis |this value must be part of the ABCD vocabulary |must be "MaterialSample" |- |Unit |*KindOfUnit |the type of sample |e.g. DNA |} ==Sample data== ===The sample identifiers=== We use identifier the same way as GBIF does. In addition to the traditional triple ID (see below) you can also provide the GUID. '''Please note that UnitID should be unique in one database!''' Mostly the extraction number is used for it. {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Unit |*<span style="color:red">SourceID</span> |short description of relevant collection |should be "DNA Bank" or "tissue collection" |- |Unit |*<span style="color:red">SourceInstitutionID</span> |short name/abbreviation of relevant institution |BGBM |- |Unit |*<span style="color:red">UnitID</span> |DNA extraction number or tissue number |DNA 123 |- |Unit |UnitGUID |sample record GUID |f6c7fd5a-2f04-4dcb-ba38-0230bf196d30 |} ===GGBN terms=== The following fields are highly recommended, some are mandatory (marked with *), but feel free to map more! <div id="wikinote">'''Please have a look at the [http://terms.tdwg.org/wiki/GGBN_Data_Standard GGBN Data Standard] documentation where all GGBN terms are described in detail. The table below is just listing the most important ones. Since the Nagoya Protocol has come into force we higly recommend to map the Permit fields. They will become mandatory for providing data to GGBN at the end of 2020. Until then all core members must have implemented the required fields in their databases.'''</div> {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- | |*RecordBasis |The specific nature of the data record, controlled vocabulary! |"PreservedSpecimen", "MaterialSample", "FossilSpecimen", "LivingSpecimen", "HumanObservation", "MachineObservation"; for Tissue or DNA samples use "MaterialSample" |- | |*materialSampleType |Classification of kind of physical sample in addition to BasisOfRecord/RecordBasis and Preparation Type |"tissue", "DNA", "specimen" |- |SpecimenUnit/Preparation |*preparationType |description of type of material, free text |or DNA: gDNA, eDNA, aDNA; for tissues/specimens: leaf, muscle, leg, blood |- |SpecimenUnit/Preparation |*preparationDate |date of DNA extraction |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparatioMaterials |extraction kit or protocoll |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparedBy |extraction staff |if unknown type "unknown" |- |SpecimenUnit/Preservation |preservation |preservation of the tissue or DNA |if unknown type "unknown" |- |SpecimenUnit/Loan/Permit |*permitStatus |Information about the presence, absence or other basic status of permits associated with the sample(s), controlled vocabulary! |Permit available, Permit not required, Permit not available, '''Unknown Material collected after 2014-10-12 cannot be in "Unknown" permit status!''' |- |SpecimenUnit/Loan/Permit |*permitStatusQualifier |Description of why a certain permit was not required or why Permit Status is unknown |"no national requirement for a permit at date of access", "officially authorized illegal holder", "collected on private land", "pre-Nagoya" |- |SpecimenUnit/Loan/Permit |*permitType |A permit is a document that allows someone to take an action that otherwise would not be allowed, controlled vocabulary! |Collecting Permit, Import Permit, Export Permit, Intellectual Property Rights, Copyright, Patent, Data use, Phytosanitary, Salvage, Exemption Permit, Material Transfer Agreement, Internationally Recognized Certificate of Compliance, Contract, Memorandum of Understanding, Memorandum of Cooperation, Veterinary Certificate, Human Pathogens, Genetically Modified Organism, Other |- |SpecimenUnit/Loan/Permit |*permitText |The text of a permit related to the gathering/shipping or further details | |- |GGBN/Amplification |amplificationDate |date of amplification; if unknown or general without content you don't have to map it |should be ISO format yyyy-mm-dd |- |GGBN/Amplification |marker | |COX1 |- |GGBN/Amplification |geneticAccessionNumber |the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD; this is a repeatable element, you can provide as much as you want |e.g. AJ45567 |- |GGBN/Amplification |genBankNumber-URI |complete link to the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD | |- |SpecimenUnit |blockedUntil |in case the sample is blocked until a specific date it's nevertheless searchable but customers cannot order it |ISO format |- |SpecimenUnit |blocked |sample is blocked (e.g. because it is consumed), but data are still available |Yes/No |- |GGBN |concentration |concentration of the DNA |65,34 |- |GGBN |@unit |unit of the concentration |µg/ml |- |GGBN |ratioOfAbsorbance260_230 |map only if filled with content |1,2 |- |GGBN |ratioOfAbsorbance260_280 |map only if filled with content |1,8 |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===The voucher identifiers=== {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Associations/UnitAssociation |*UnitID |the UnitID or CatalogueNumber used for GBIF |e.g. the barcode number of your specimens |- |Associations/UnitAssociation |*SourceInstitutionID |the SourceInstitutionID or InstitutionCode used for GBIF |e.g. the acronym of your institution |- |Associations/UnitAssociation |*SourceName |the SourceID or CollectionCode used for GBIF |e.g. the name of the collection where the specimen belongs to, e.g. "Birds" |- |Associations/UnitAssociation |*AssociationType |the Relation between the DNA and the voucher |e.g. "DNA and voucher from same individual" |- |Associations/UnitAssociation |*DatasetAccessPoint |the wrapper url of the voucher record |e.g. "http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar" |} ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} ===Multimedia items of the voucher=== These should be mapped in the specimen mapping, not the DNA mapping. {| class="wikitable" ! Category ! Element ! Discription ! Comment |- | rowspan="3" | Metadata | Title | | |- | IPR | | |- | TechnicalContact | | |- | Unit | UnitID | | |} ed8124c88fd7f1fbd25a1014b8bde2061a0e82ea 992 991 2018-09-27T09:42:17Z JoergHoletschek 3 /* Metadata */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. {| class="wikitable" ! width="120px" | Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone |john.doe@acme.com |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone |john.doe@acme.com |- |rowspan="2" | RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |Description/Representation |*Details |Short text to describe the focus and number of samples in your DNA collection. '''It should include the following phrase:''' ''The DNA bank is part of the Global Genome Biodiversity Network (GGBN) which was founded in 2011. The network provides a technically optimized DNA collection service facility for all biological research accessible via one central web portal. The network promotes deposition of well documented reference DNA samples after project completion or data publication from scientists of other universities and institutions.'' |The DNA bank of the Botanic Garden and Botanical Museum Berlin-Dahlem holds currently a collection of 20000 plant DNA and tissue samples growing constantly. Its core collection focuses on the flora of Germany but it comprises botanical samples collected worldwide. |- |Description/Representation |*@language |language of description |should be "en" as literal |- |Description/Representation |*Title |Short title that describes your DNA collection |DNA Bank of the Herbarium Berolinense |- |IPRStatements/Citation |*Text | |Droege, G. (Ed.) 2008 - (continuously updated): DNA samples of the DNA bank at the BGBM (Botanic Garden and Botanical Museum Berlin-Dahlem). |- |IPRStatements/Citation |*@language |language of citation |should be "en" as literal |- |IPRStatements/Copyright |*Text | |The copyright for any material created by the DNA bank of the BGBM is reserved. The duplication or use of information and data such as texts or images is only permitted with the indication of the source or with prior approval by the BGBM. |- |IPRStatements/Copyright |*@language |language of copyright |should be "en" as literal |- |IPRStatements/TermsOfUse |*Text | |The use of the data is allowed only for non-profit scientific use and for non-profit nature conservation purpose. The data bases or part of it may only be used or copied by the written permission from the legal owner. |- |IPRStatements/TermsOfUse |*@language |language of the terms of use |should be "en" as literal |- | |IconURI |complete url path to the logo of your institution | |- |Owner |*Address |Complete Address of the institution that owns the DNA bank samples and data | |- |Owner/Representation |*Text |name of your institution |Botanic Garden an Botanical Museum Berlin-Dahlem (Freie Universität Berlin) |- |Owner |*URL |path to website of your institution |http://www.bgbm.org |- |Unit |*RecordBasis |this value must be part of the ABCD vocabulary |must be "MaterialSample" |- |Unit |*KindOfUnit |the type of sample |e.g. DNA |} ==Sample data== ===The sample identifiers=== We use identifier the same way as GBIF does. In addition to the traditional triple ID (see below) you can also provide the GUID. '''Please note that UnitID should be unique in one database!''' Mostly the extraction number is used for it. {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Unit |*<span style="color:red">SourceID</span> |short description of relevant collection |should be "DNA Bank" or "tissue collection" |- |Unit |*<span style="color:red">SourceInstitutionID</span> |short name/abbreviation of relevant institution |BGBM |- |Unit |*<span style="color:red">UnitID</span> |DNA extraction number or tissue number |DNA 123 |- |Unit |UnitGUID |sample record GUID |f6c7fd5a-2f04-4dcb-ba38-0230bf196d30 |} ===GGBN terms=== The following fields are highly recommended, some are mandatory (marked with *), but feel free to map more! <div id="wikinote">'''Please have a look at the [http://terms.tdwg.org/wiki/GGBN_Data_Standard GGBN Data Standard] documentation where all GGBN terms are described in detail. The table below is just listing the most important ones. Since the Nagoya Protocol has come into force we higly recommend to map the Permit fields. They will become mandatory for providing data to GGBN at the end of 2020. Until then all core members must have implemented the required fields in their databases.'''</div> {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- | |*RecordBasis |The specific nature of the data record, controlled vocabulary! |"PreservedSpecimen", "MaterialSample", "FossilSpecimen", "LivingSpecimen", "HumanObservation", "MachineObservation"; for Tissue or DNA samples use "MaterialSample" |- | |*materialSampleType |Classification of kind of physical sample in addition to BasisOfRecord/RecordBasis and Preparation Type |"tissue", "DNA", "specimen" |- |SpecimenUnit/Preparation |*preparationType |description of type of material, free text |or DNA: gDNA, eDNA, aDNA; for tissues/specimens: leaf, muscle, leg, blood |- |SpecimenUnit/Preparation |*preparationDate |date of DNA extraction |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparatioMaterials |extraction kit or protocoll |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparedBy |extraction staff |if unknown type "unknown" |- |SpecimenUnit/Preservation |preservation |preservation of the tissue or DNA |if unknown type "unknown" |- |SpecimenUnit/Loan/Permit |*permitStatus |Information about the presence, absence or other basic status of permits associated with the sample(s), controlled vocabulary! |Permit available, Permit not required, Permit not available, '''Unknown Material collected after 2014-10-12 cannot be in "Unknown" permit status!''' |- |SpecimenUnit/Loan/Permit |*permitStatusQualifier |Description of why a certain permit was not required or why Permit Status is unknown |"no national requirement for a permit at date of access", "officially authorized illegal holder", "collected on private land", "pre-Nagoya" |- |SpecimenUnit/Loan/Permit |*permitType |A permit is a document that allows someone to take an action that otherwise would not be allowed, controlled vocabulary! |Collecting Permit, Import Permit, Export Permit, Intellectual Property Rights, Copyright, Patent, Data use, Phytosanitary, Salvage, Exemption Permit, Material Transfer Agreement, Internationally Recognized Certificate of Compliance, Contract, Memorandum of Understanding, Memorandum of Cooperation, Veterinary Certificate, Human Pathogens, Genetically Modified Organism, Other |- |SpecimenUnit/Loan/Permit |*permitText |The text of a permit related to the gathering/shipping or further details | |- |GGBN/Amplification |amplificationDate |date of amplification; if unknown or general without content you don't have to map it |should be ISO format yyyy-mm-dd |- |GGBN/Amplification |marker | |COX1 |- |GGBN/Amplification |geneticAccessionNumber |the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD; this is a repeatable element, you can provide as much as you want |e.g. AJ45567 |- |GGBN/Amplification |genBankNumber-URI |complete link to the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD | |- |SpecimenUnit |blockedUntil |in case the sample is blocked until a specific date it's nevertheless searchable but customers cannot order it |ISO format |- |SpecimenUnit |blocked |sample is blocked (e.g. because it is consumed), but data are still available |Yes/No |- |GGBN |concentration |concentration of the DNA |65,34 |- |GGBN |@unit |unit of the concentration |µg/ml |- |GGBN |ratioOfAbsorbance260_230 |map only if filled with content |1,2 |- |GGBN |ratioOfAbsorbance260_280 |map only if filled with content |1,8 |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===The voucher identifiers=== {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Associations/UnitAssociation |*UnitID |the UnitID or CatalogueNumber used for GBIF |e.g. the barcode number of your specimens |- |Associations/UnitAssociation |*SourceInstitutionID |the SourceInstitutionID or InstitutionCode used for GBIF |e.g. the acronym of your institution |- |Associations/UnitAssociation |*SourceName |the SourceID or CollectionCode used for GBIF |e.g. the name of the collection where the specimen belongs to, e.g. "Birds" |- |Associations/UnitAssociation |*AssociationType |the Relation between the DNA and the voucher |e.g. "DNA and voucher from same individual" |- |Associations/UnitAssociation |*DatasetAccessPoint |the wrapper url of the voucher record |e.g. "http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar" |} ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} ===Multimedia items of the voucher=== These should be mapped in the specimen mapping, not the DNA mapping. {| class="wikitable" ! Category ! Element ! Discription ! Comment |- | rowspan="3" | Metadata | Title | | |- | IPR | | |- | TechnicalContact | | |- | Unit | UnitID | | |} e91df880126612b6c103efb72c6c9c66acf79680 993 992 2018-09-27T09:54:00Z JoergHoletschek 3 /* Metadata */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. {| class="wikitable" ! width="120px" | Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone |john.doe@acme.com |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone |john.doe@acme.com |- |rowspan="2" | RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |IPRStatements/Citation |*Text | |Droege, G. (Ed.) 2008 - (continuously updated): DNA samples of the DNA bank at the BGBM (Botanic Garden and Botanical Museum Berlin-Dahlem). |- |IPRStatements/Citation |*@language |language of citation |should be "en" as literal |- |IPRStatements/Copyright |*Text | |The copyright for any material created by the DNA bank of the BGBM is reserved. The duplication or use of information and data such as texts or images is only permitted with the indication of the source or with prior approval by the BGBM. |- |IPRStatements/Copyright |*@language |language of copyright |should be "en" as literal |- |IPRStatements/TermsOfUse |*Text | |The use of the data is allowed only for non-profit scientific use and for non-profit nature conservation purpose. The data bases or part of it may only be used or copied by the written permission from the legal owner. |- |IPRStatements/TermsOfUse |*@language |language of the terms of use |should be "en" as literal |- | |IconURI |complete url path to the logo of your institution | |- |Owner |*Address |Complete Address of the institution that owns the DNA bank samples and data | |- |Owner/Representation |*Text |name of your institution |Botanic Garden an Botanical Museum Berlin-Dahlem (Freie Universität Berlin) |- |Owner |*URL |path to website of your institution |http://www.bgbm.org |- |Unit |*RecordBasis |this value must be part of the ABCD vocabulary |must be "MaterialSample" |- |Unit |*KindOfUnit |the type of sample |e.g. DNA |} ==Sample data== ===The sample identifiers=== We use identifier the same way as GBIF does. In addition to the traditional triple ID (see below) you can also provide the GUID. '''Please note that UnitID should be unique in one database!''' Mostly the extraction number is used for it. {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Unit |*<span style="color:red">SourceID</span> |short description of relevant collection |should be "DNA Bank" or "tissue collection" |- |Unit |*<span style="color:red">SourceInstitutionID</span> |short name/abbreviation of relevant institution |BGBM |- |Unit |*<span style="color:red">UnitID</span> |DNA extraction number or tissue number |DNA 123 |- |Unit |UnitGUID |sample record GUID |f6c7fd5a-2f04-4dcb-ba38-0230bf196d30 |} ===GGBN terms=== The following fields are highly recommended, some are mandatory (marked with *), but feel free to map more! <div id="wikinote">'''Please have a look at the [http://terms.tdwg.org/wiki/GGBN_Data_Standard GGBN Data Standard] documentation where all GGBN terms are described in detail. The table below is just listing the most important ones. Since the Nagoya Protocol has come into force we higly recommend to map the Permit fields. They will become mandatory for providing data to GGBN at the end of 2020. Until then all core members must have implemented the required fields in their databases.'''</div> {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- | |*RecordBasis |The specific nature of the data record, controlled vocabulary! |"PreservedSpecimen", "MaterialSample", "FossilSpecimen", "LivingSpecimen", "HumanObservation", "MachineObservation"; for Tissue or DNA samples use "MaterialSample" |- | |*materialSampleType |Classification of kind of physical sample in addition to BasisOfRecord/RecordBasis and Preparation Type |"tissue", "DNA", "specimen" |- |SpecimenUnit/Preparation |*preparationType |description of type of material, free text |or DNA: gDNA, eDNA, aDNA; for tissues/specimens: leaf, muscle, leg, blood |- |SpecimenUnit/Preparation |*preparationDate |date of DNA extraction |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparatioMaterials |extraction kit or protocoll |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparedBy |extraction staff |if unknown type "unknown" |- |SpecimenUnit/Preservation |preservation |preservation of the tissue or DNA |if unknown type "unknown" |- |SpecimenUnit/Loan/Permit |*permitStatus |Information about the presence, absence or other basic status of permits associated with the sample(s), controlled vocabulary! |Permit available, Permit not required, Permit not available, '''Unknown Material collected after 2014-10-12 cannot be in "Unknown" permit status!''' |- |SpecimenUnit/Loan/Permit |*permitStatusQualifier |Description of why a certain permit was not required or why Permit Status is unknown |"no national requirement for a permit at date of access", "officially authorized illegal holder", "collected on private land", "pre-Nagoya" |- |SpecimenUnit/Loan/Permit |*permitType |A permit is a document that allows someone to take an action that otherwise would not be allowed, controlled vocabulary! |Collecting Permit, Import Permit, Export Permit, Intellectual Property Rights, Copyright, Patent, Data use, Phytosanitary, Salvage, Exemption Permit, Material Transfer Agreement, Internationally Recognized Certificate of Compliance, Contract, Memorandum of Understanding, Memorandum of Cooperation, Veterinary Certificate, Human Pathogens, Genetically Modified Organism, Other |- |SpecimenUnit/Loan/Permit |*permitText |The text of a permit related to the gathering/shipping or further details | |- |GGBN/Amplification |amplificationDate |date of amplification; if unknown or general without content you don't have to map it |should be ISO format yyyy-mm-dd |- |GGBN/Amplification |marker | |COX1 |- |GGBN/Amplification |geneticAccessionNumber |the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD; this is a repeatable element, you can provide as much as you want |e.g. AJ45567 |- |GGBN/Amplification |genBankNumber-URI |complete link to the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD | |- |SpecimenUnit |blockedUntil |in case the sample is blocked until a specific date it's nevertheless searchable but customers cannot order it |ISO format |- |SpecimenUnit |blocked |sample is blocked (e.g. because it is consumed), but data are still available |Yes/No |- |GGBN |concentration |concentration of the DNA |65,34 |- |GGBN |@unit |unit of the concentration |µg/ml |- |GGBN |ratioOfAbsorbance260_230 |map only if filled with content |1,2 |- |GGBN |ratioOfAbsorbance260_280 |map only if filled with content |1,8 |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===The voucher identifiers=== {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Associations/UnitAssociation |*UnitID |the UnitID or CatalogueNumber used for GBIF |e.g. the barcode number of your specimens |- |Associations/UnitAssociation |*SourceInstitutionID |the SourceInstitutionID or InstitutionCode used for GBIF |e.g. the acronym of your institution |- |Associations/UnitAssociation |*SourceName |the SourceID or CollectionCode used for GBIF |e.g. the name of the collection where the specimen belongs to, e.g. "Birds" |- |Associations/UnitAssociation |*AssociationType |the Relation between the DNA and the voucher |e.g. "DNA and voucher from same individual" |- |Associations/UnitAssociation |*DatasetAccessPoint |the wrapper url of the voucher record |e.g. "http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar" |} ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} ===Multimedia items of the voucher=== These should be mapped in the specimen mapping, not the DNA mapping. {| class="wikitable" ! Category ! Element ! Discription ! Comment |- | rowspan="3" | Metadata | Title | | |- | IPR | | |- | TechnicalContact | | |- | Unit | UnitID | | |} 91b411cb3002d5b90a96be220b87217d1849da48 994 993 2018-09-27T09:56:50Z JoergHoletschek 3 /* Metadata */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. {| class="wikitable" ! width="120px" | Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone |john.doe@acme.com |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone |john.doe@acme.com |- |rowspan="2" | RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |IPRStatements/Citation |*Text | |Droege, G. (Ed.) 2008 - (continuously updated): DNA samples of the DNA bank at the BGBM (Botanic Garden and Botanical Museum Berlin-Dahlem). |- |IPRStatements/Citation |*@language |language of citation |should be "en" as literal |- |IPRStatements/Copyright |*Text | |The copyright for any material created by the DNA bank of the BGBM is reserved. The duplication or use of information and data such as texts or images is only permitted with the indication of the source or with prior approval by the BGBM. |- |IPRStatements/Copyright |*@language |language of copyright |should be "en" as literal |- |IPRStatements/TermsOfUse |*Text | |The use of the data is allowed only for non-profit scientific use and for non-profit nature conservation purpose. The data bases or part of it may only be used or copied by the written permission from the legal owner. |- |IPRStatements/TermsOfUse |*@language |language of the terms of use |should be "en" as literal |- | |IconURI |complete url path to the logo of your institution | |- |Owner |*Address |Complete Address of the institution that owns the DNA bank samples and data | |- |Owner/Representation |*Text |name of your institution |Botanic Garden an Botanical Museum Berlin-Dahlem (Freie Universität Berlin) |- |Owner |*URL |path to website of your institution |http://www.bgbm.org |- |Unit |*RecordBasis |this value must be part of the ABCD vocabulary |must be "MaterialSample" |- |Unit |*KindOfUnit |the type of sample |e.g. DNA |} ==Sample data== ===The sample identifiers=== We use identifier the same way as GBIF does. In addition to the traditional triple ID (see below) you can also provide the GUID. '''Please note that UnitID should be unique in one database!''' Mostly the extraction number is used for it. {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Unit |*<span style="color:red">SourceID</span> |short description of relevant collection |should be "DNA Bank" or "tissue collection" |- |Unit |*<span style="color:red">SourceInstitutionID</span> |short name/abbreviation of relevant institution |BGBM |- |Unit |*<span style="color:red">UnitID</span> |DNA extraction number or tissue number |DNA 123 |- |Unit |UnitGUID |sample record GUID |f6c7fd5a-2f04-4dcb-ba38-0230bf196d30 |} ===GGBN terms=== The following fields are highly recommended, some are mandatory (marked with *), but feel free to map more! <div id="wikinote">'''Please have a look at the [http://terms.tdwg.org/wiki/GGBN_Data_Standard GGBN Data Standard] documentation where all GGBN terms are described in detail. The table below is just listing the most important ones. Since the Nagoya Protocol has come into force we higly recommend to map the Permit fields. They will become mandatory for providing data to GGBN at the end of 2020. Until then all core members must have implemented the required fields in their databases.'''</div> {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- | |*RecordBasis |The specific nature of the data record, controlled vocabulary! |"PreservedSpecimen", "MaterialSample", "FossilSpecimen", "LivingSpecimen", "HumanObservation", "MachineObservation"; for Tissue or DNA samples use "MaterialSample" |- | |*materialSampleType |Classification of kind of physical sample in addition to BasisOfRecord/RecordBasis and Preparation Type |"tissue", "DNA", "specimen" |- |SpecimenUnit/Preparation |*preparationType |description of type of material, free text |or DNA: gDNA, eDNA, aDNA; for tissues/specimens: leaf, muscle, leg, blood |- |SpecimenUnit/Preparation |*preparationDate |date of DNA extraction |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparatioMaterials |extraction kit or protocoll |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparedBy |extraction staff |if unknown type "unknown" |- |SpecimenUnit/Preservation |preservation |preservation of the tissue or DNA |if unknown type "unknown" |- |SpecimenUnit/Loan/Permit |*permitStatus |Information about the presence, absence or other basic status of permits associated with the sample(s), controlled vocabulary! |Permit available, Permit not required, Permit not available, '''Unknown Material collected after 2014-10-12 cannot be in "Unknown" permit status!''' |- |SpecimenUnit/Loan/Permit |*permitStatusQualifier |Description of why a certain permit was not required or why Permit Status is unknown |"no national requirement for a permit at date of access", "officially authorized illegal holder", "collected on private land", "pre-Nagoya" |- |SpecimenUnit/Loan/Permit |*permitType |A permit is a document that allows someone to take an action that otherwise would not be allowed, controlled vocabulary! |Collecting Permit, Import Permit, Export Permit, Intellectual Property Rights, Copyright, Patent, Data use, Phytosanitary, Salvage, Exemption Permit, Material Transfer Agreement, Internationally Recognized Certificate of Compliance, Contract, Memorandum of Understanding, Memorandum of Cooperation, Veterinary Certificate, Human Pathogens, Genetically Modified Organism, Other |- |SpecimenUnit/Loan/Permit |*permitText |The text of a permit related to the gathering/shipping or further details | |- |GGBN/Amplification |amplificationDate |date of amplification; if unknown or general without content you don't have to map it |should be ISO format yyyy-mm-dd |- |GGBN/Amplification |marker | |COX1 |- |GGBN/Amplification |geneticAccessionNumber |the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD; this is a repeatable element, you can provide as much as you want |e.g. AJ45567 |- |GGBN/Amplification |genBankNumber-URI |complete link to the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD | |- |SpecimenUnit |blockedUntil |in case the sample is blocked until a specific date it's nevertheless searchable but customers cannot order it |ISO format |- |SpecimenUnit |blocked |sample is blocked (e.g. because it is consumed), but data are still available |Yes/No |- |GGBN |concentration |concentration of the DNA |65,34 |- |GGBN |@unit |unit of the concentration |µg/ml |- |GGBN |ratioOfAbsorbance260_230 |map only if filled with content |1,2 |- |GGBN |ratioOfAbsorbance260_280 |map only if filled with content |1,8 |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===The voucher identifiers=== {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Associations/UnitAssociation |*UnitID |the UnitID or CatalogueNumber used for GBIF |e.g. the barcode number of your specimens |- |Associations/UnitAssociation |*SourceInstitutionID |the SourceInstitutionID or InstitutionCode used for GBIF |e.g. the acronym of your institution |- |Associations/UnitAssociation |*SourceName |the SourceID or CollectionCode used for GBIF |e.g. the name of the collection where the specimen belongs to, e.g. "Birds" |- |Associations/UnitAssociation |*AssociationType |the Relation between the DNA and the voucher |e.g. "DNA and voucher from same individual" |- |Associations/UnitAssociation |*DatasetAccessPoint |the wrapper url of the voucher record |e.g. "http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar" |} ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} ===Multimedia items of the voucher=== These should be mapped in the specimen mapping, not the DNA mapping. {| class="wikitable" ! Category ! Element ! Discription ! Comment |- | rowspan="3" | Metadata | Title | | |- | IPR | | |- | TechnicalContact | | |- | Unit | UnitID | | |} 514d8657918848a059aa3460a56e4ce4fc391194 995 994 2018-09-27T10:00:14Z JoergHoletschek 3 /* Metadata */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. {| class="wikitable" ! width="120px" | Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone |john.doe@acme.com |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone |john.doe@acme.com |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |IPRStatements/Citation |*Text | |Droege, G. (Ed.) 2008 - (continuously updated): DNA samples of the DNA bank at the BGBM (Botanic Garden and Botanical Museum Berlin-Dahlem). |- |IPRStatements/Citation |*@language |language of citation |should be "en" as literal |- |IPRStatements/Copyright |*Text | |The copyright for any material created by the DNA bank of the BGBM is reserved. The duplication or use of information and data such as texts or images is only permitted with the indication of the source or with prior approval by the BGBM. |- |IPRStatements/Copyright |*@language |language of copyright |should be "en" as literal |- |IPRStatements/TermsOfUse |*Text | |The use of the data is allowed only for non-profit scientific use and for non-profit nature conservation purpose. The data bases or part of it may only be used or copied by the written permission from the legal owner. |- |IPRStatements/TermsOfUse |*@language |language of the terms of use |should be "en" as literal |- | |IconURI |complete url path to the logo of your institution | |- |Owner |*Address |Complete Address of the institution that owns the DNA bank samples and data | |- |Owner/Representation |*Text |name of your institution |Botanic Garden an Botanical Museum Berlin-Dahlem (Freie Universität Berlin) |- |Owner |*URL |path to website of your institution |http://www.bgbm.org |- |Unit |*RecordBasis |this value must be part of the ABCD vocabulary |must be "MaterialSample" |- |Unit |*KindOfUnit |the type of sample |e.g. DNA |} ==Sample data== ===The sample identifiers=== We use identifier the same way as GBIF does. In addition to the traditional triple ID (see below) you can also provide the GUID. '''Please note that UnitID should be unique in one database!''' Mostly the extraction number is used for it. {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Unit |*<span style="color:red">SourceID</span> |short description of relevant collection |should be "DNA Bank" or "tissue collection" |- |Unit |*<span style="color:red">SourceInstitutionID</span> |short name/abbreviation of relevant institution |BGBM |- |Unit |*<span style="color:red">UnitID</span> |DNA extraction number or tissue number |DNA 123 |- |Unit |UnitGUID |sample record GUID |f6c7fd5a-2f04-4dcb-ba38-0230bf196d30 |} ===GGBN terms=== The following fields are highly recommended, some are mandatory (marked with *), but feel free to map more! <div id="wikinote">'''Please have a look at the [http://terms.tdwg.org/wiki/GGBN_Data_Standard GGBN Data Standard] documentation where all GGBN terms are described in detail. The table below is just listing the most important ones. Since the Nagoya Protocol has come into force we higly recommend to map the Permit fields. They will become mandatory for providing data to GGBN at the end of 2020. Until then all core members must have implemented the required fields in their databases.'''</div> {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- | |*RecordBasis |The specific nature of the data record, controlled vocabulary! |"PreservedSpecimen", "MaterialSample", "FossilSpecimen", "LivingSpecimen", "HumanObservation", "MachineObservation"; for Tissue or DNA samples use "MaterialSample" |- | |*materialSampleType |Classification of kind of physical sample in addition to BasisOfRecord/RecordBasis and Preparation Type |"tissue", "DNA", "specimen" |- |SpecimenUnit/Preparation |*preparationType |description of type of material, free text |or DNA: gDNA, eDNA, aDNA; for tissues/specimens: leaf, muscle, leg, blood |- |SpecimenUnit/Preparation |*preparationDate |date of DNA extraction |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparatioMaterials |extraction kit or protocoll |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparedBy |extraction staff |if unknown type "unknown" |- |SpecimenUnit/Preservation |preservation |preservation of the tissue or DNA |if unknown type "unknown" |- |SpecimenUnit/Loan/Permit |*permitStatus |Information about the presence, absence or other basic status of permits associated with the sample(s), controlled vocabulary! |Permit available, Permit not required, Permit not available, '''Unknown Material collected after 2014-10-12 cannot be in "Unknown" permit status!''' |- |SpecimenUnit/Loan/Permit |*permitStatusQualifier |Description of why a certain permit was not required or why Permit Status is unknown |"no national requirement for a permit at date of access", "officially authorized illegal holder", "collected on private land", "pre-Nagoya" |- |SpecimenUnit/Loan/Permit |*permitType |A permit is a document that allows someone to take an action that otherwise would not be allowed, controlled vocabulary! |Collecting Permit, Import Permit, Export Permit, Intellectual Property Rights, Copyright, Patent, Data use, Phytosanitary, Salvage, Exemption Permit, Material Transfer Agreement, Internationally Recognized Certificate of Compliance, Contract, Memorandum of Understanding, Memorandum of Cooperation, Veterinary Certificate, Human Pathogens, Genetically Modified Organism, Other |- |SpecimenUnit/Loan/Permit |*permitText |The text of a permit related to the gathering/shipping or further details | |- |GGBN/Amplification |amplificationDate |date of amplification; if unknown or general without content you don't have to map it |should be ISO format yyyy-mm-dd |- |GGBN/Amplification |marker | |COX1 |- |GGBN/Amplification |geneticAccessionNumber |the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD; this is a repeatable element, you can provide as much as you want |e.g. AJ45567 |- |GGBN/Amplification |genBankNumber-URI |complete link to the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD | |- |SpecimenUnit |blockedUntil |in case the sample is blocked until a specific date it's nevertheless searchable but customers cannot order it |ISO format |- |SpecimenUnit |blocked |sample is blocked (e.g. because it is consumed), but data are still available |Yes/No |- |GGBN |concentration |concentration of the DNA |65,34 |- |GGBN |@unit |unit of the concentration |µg/ml |- |GGBN |ratioOfAbsorbance260_230 |map only if filled with content |1,2 |- |GGBN |ratioOfAbsorbance260_280 |map only if filled with content |1,8 |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===The voucher identifiers=== {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Associations/UnitAssociation |*UnitID |the UnitID or CatalogueNumber used for GBIF |e.g. the barcode number of your specimens |- |Associations/UnitAssociation |*SourceInstitutionID |the SourceInstitutionID or InstitutionCode used for GBIF |e.g. the acronym of your institution |- |Associations/UnitAssociation |*SourceName |the SourceID or CollectionCode used for GBIF |e.g. the name of the collection where the specimen belongs to, e.g. "Birds" |- |Associations/UnitAssociation |*AssociationType |the Relation between the DNA and the voucher |e.g. "DNA and voucher from same individual" |- |Associations/UnitAssociation |*DatasetAccessPoint |the wrapper url of the voucher record |e.g. "http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar" |} ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} ===Multimedia items of the voucher=== These should be mapped in the specimen mapping, not the DNA mapping. {| class="wikitable" ! Category ! Element ! Discription ! Comment |- | rowspan="3" | Metadata | Title | | |- | IPR | | |- | TechnicalContact | | |- | Unit | UnitID | | |} be340722c9c4fae5c7cf52d31fc6f83f5c097be6 996 995 2018-09-27T10:30:17Z JoergHoletschek 3 /* Metadata */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. {| class="wikitable" ! width="120px" | Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone |+49 30 838 99999999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone |+49 30 838 99999999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="4" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="4" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber |+49 30 838 99999999 |- |URIs/URL |https://www.bgbm.org |- |IPRStatements/Citation |*Text | |Droege, G. (Ed.) 2008 - (continuously updated): DNA samples of the DNA bank at the BGBM (Botanic Garden and Botanical Museum Berlin-Dahlem). |- |IPRStatements/Citation |*@language |language of citation |should be "en" as literal |- |IPRStatements/Copyright |*Text | |The copyright for any material created by the DNA bank of the BGBM is reserved. The duplication or use of information and data such as texts or images is only permitted with the indication of the source or with prior approval by the BGBM. |- |IPRStatements/Copyright |*@language |language of copyright |should be "en" as literal |- |IPRStatements/TermsOfUse |*Text | |The use of the data is allowed only for non-profit scientific use and for non-profit nature conservation purpose. The data bases or part of it may only be used or copied by the written permission from the legal owner. |- |IPRStatements/TermsOfUse |*@language |language of the terms of use |should be "en" as literal |- | |IconURI |complete url path to the logo of your institution | |- |Owner |*Address |Complete Address of the institution that owns the DNA bank samples and data | |- |Owner/Representation |*Text |name of your institution |Botanic Garden an Botanical Museum Berlin-Dahlem (Freie Universität Berlin) |- |Owner |*URL |path to website of your institution |http://www.bgbm.org |- |Unit |*RecordBasis |this value must be part of the ABCD vocabulary |must be "MaterialSample" |- |Unit |*KindOfUnit |the type of sample |e.g. DNA |} ==Sample data== ===The sample identifiers=== We use identifier the same way as GBIF does. In addition to the traditional triple ID (see below) you can also provide the GUID. '''Please note that UnitID should be unique in one database!''' Mostly the extraction number is used for it. {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Unit |*<span style="color:red">SourceID</span> |short description of relevant collection |should be "DNA Bank" or "tissue collection" |- |Unit |*<span style="color:red">SourceInstitutionID</span> |short name/abbreviation of relevant institution |BGBM |- |Unit |*<span style="color:red">UnitID</span> |DNA extraction number or tissue number |DNA 123 |- |Unit |UnitGUID |sample record GUID |f6c7fd5a-2f04-4dcb-ba38-0230bf196d30 |} ===GGBN terms=== The following fields are highly recommended, some are mandatory (marked with *), but feel free to map more! <div id="wikinote">'''Please have a look at the [http://terms.tdwg.org/wiki/GGBN_Data_Standard GGBN Data Standard] documentation where all GGBN terms are described in detail. The table below is just listing the most important ones. Since the Nagoya Protocol has come into force we higly recommend to map the Permit fields. They will become mandatory for providing data to GGBN at the end of 2020. Until then all core members must have implemented the required fields in their databases.'''</div> {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- | |*RecordBasis |The specific nature of the data record, controlled vocabulary! |"PreservedSpecimen", "MaterialSample", "FossilSpecimen", "LivingSpecimen", "HumanObservation", "MachineObservation"; for Tissue or DNA samples use "MaterialSample" |- | |*materialSampleType |Classification of kind of physical sample in addition to BasisOfRecord/RecordBasis and Preparation Type |"tissue", "DNA", "specimen" |- |SpecimenUnit/Preparation |*preparationType |description of type of material, free text |or DNA: gDNA, eDNA, aDNA; for tissues/specimens: leaf, muscle, leg, blood |- |SpecimenUnit/Preparation |*preparationDate |date of DNA extraction |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparatioMaterials |extraction kit or protocoll |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparedBy |extraction staff |if unknown type "unknown" |- |SpecimenUnit/Preservation |preservation |preservation of the tissue or DNA |if unknown type "unknown" |- |SpecimenUnit/Loan/Permit |*permitStatus |Information about the presence, absence or other basic status of permits associated with the sample(s), controlled vocabulary! |Permit available, Permit not required, Permit not available, '''Unknown Material collected after 2014-10-12 cannot be in "Unknown" permit status!''' |- |SpecimenUnit/Loan/Permit |*permitStatusQualifier |Description of why a certain permit was not required or why Permit Status is unknown |"no national requirement for a permit at date of access", "officially authorized illegal holder", "collected on private land", "pre-Nagoya" |- |SpecimenUnit/Loan/Permit |*permitType |A permit is a document that allows someone to take an action that otherwise would not be allowed, controlled vocabulary! |Collecting Permit, Import Permit, Export Permit, Intellectual Property Rights, Copyright, Patent, Data use, Phytosanitary, Salvage, Exemption Permit, Material Transfer Agreement, Internationally Recognized Certificate of Compliance, Contract, Memorandum of Understanding, Memorandum of Cooperation, Veterinary Certificate, Human Pathogens, Genetically Modified Organism, Other |- |SpecimenUnit/Loan/Permit |*permitText |The text of a permit related to the gathering/shipping or further details | |- |GGBN/Amplification |amplificationDate |date of amplification; if unknown or general without content you don't have to map it |should be ISO format yyyy-mm-dd |- |GGBN/Amplification |marker | |COX1 |- |GGBN/Amplification |geneticAccessionNumber |the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD; this is a repeatable element, you can provide as much as you want |e.g. AJ45567 |- |GGBN/Amplification |genBankNumber-URI |complete link to the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD | |- |SpecimenUnit |blockedUntil |in case the sample is blocked until a specific date it's nevertheless searchable but customers cannot order it |ISO format |- |SpecimenUnit |blocked |sample is blocked (e.g. because it is consumed), but data are still available |Yes/No |- |GGBN |concentration |concentration of the DNA |65,34 |- |GGBN |@unit |unit of the concentration |µg/ml |- |GGBN |ratioOfAbsorbance260_230 |map only if filled with content |1,2 |- |GGBN |ratioOfAbsorbance260_280 |map only if filled with content |1,8 |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===The voucher identifiers=== {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Associations/UnitAssociation |*UnitID |the UnitID or CatalogueNumber used for GBIF |e.g. the barcode number of your specimens |- |Associations/UnitAssociation |*SourceInstitutionID |the SourceInstitutionID or InstitutionCode used for GBIF |e.g. the acronym of your institution |- |Associations/UnitAssociation |*SourceName |the SourceID or CollectionCode used for GBIF |e.g. the name of the collection where the specimen belongs to, e.g. "Birds" |- |Associations/UnitAssociation |*AssociationType |the Relation between the DNA and the voucher |e.g. "DNA and voucher from same individual" |- |Associations/UnitAssociation |*DatasetAccessPoint |the wrapper url of the voucher record |e.g. "http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar" |} ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} ===Multimedia items of the voucher=== These should be mapped in the specimen mapping, not the DNA mapping. {| class="wikitable" ! Category ! Element ! Discription ! Comment |- | rowspan="3" | Metadata | Title | | |- | IPR | | |- | TechnicalContact | | |- | Unit | UnitID | | |} 16b6a2eee81e35c2d4bf8ef32ad4cfd396e1b6d4 997 996 2018-09-27T10:39:38Z JoergHoletschek 3 /* Metadata */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. {| class="wikitable" ! width="120px" | Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="4" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="4" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999999 |- |URIs/URL |https://www.bgbm.org |- |IPRStatements/Citation |*Text | |Droege, G. (Ed.) 2008 - (continuously updated): DNA samples of the DNA bank at the BGBM (Botanic Garden and Botanical Museum Berlin-Dahlem). |- |IPRStatements/Citation |*@language |language of citation |should be "en" as literal |- |IPRStatements/Copyright |*Text | |The copyright for any material created by the DNA bank of the BGBM is reserved. The duplication or use of information and data such as texts or images is only permitted with the indication of the source or with prior approval by the BGBM. |- |IPRStatements/Copyright |*@language |language of copyright |should be "en" as literal |- |IPRStatements/TermsOfUse |*Text | |The use of the data is allowed only for non-profit scientific use and for non-profit nature conservation purpose. The data bases or part of it may only be used or copied by the written permission from the legal owner. |- |IPRStatements/TermsOfUse |*@language |language of the terms of use |should be "en" as literal |- | |IconURI |complete url path to the logo of your institution | |- |Owner |*Address |Complete Address of the institution that owns the DNA bank samples and data | |- |Owner/Representation |*Text |name of your institution |Botanic Garden an Botanical Museum Berlin-Dahlem (Freie Universität Berlin) |- |Owner |*URL |path to website of your institution |http://www.bgbm.org |- |Unit |*RecordBasis |this value must be part of the ABCD vocabulary |must be "MaterialSample" |- |Unit |*KindOfUnit |the type of sample |e.g. DNA |} ==Sample data== ===The sample identifiers=== We use identifier the same way as GBIF does. In addition to the traditional triple ID (see below) you can also provide the GUID. '''Please note that UnitID should be unique in one database!''' Mostly the extraction number is used for it. {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Unit |*<span style="color:red">SourceID</span> |short description of relevant collection |should be "DNA Bank" or "tissue collection" |- |Unit |*<span style="color:red">SourceInstitutionID</span> |short name/abbreviation of relevant institution |BGBM |- |Unit |*<span style="color:red">UnitID</span> |DNA extraction number or tissue number |DNA 123 |- |Unit |UnitGUID |sample record GUID |f6c7fd5a-2f04-4dcb-ba38-0230bf196d30 |} ===GGBN terms=== The following fields are highly recommended, some are mandatory (marked with *), but feel free to map more! <div id="wikinote">'''Please have a look at the [http://terms.tdwg.org/wiki/GGBN_Data_Standard GGBN Data Standard] documentation where all GGBN terms are described in detail. The table below is just listing the most important ones. Since the Nagoya Protocol has come into force we higly recommend to map the Permit fields. They will become mandatory for providing data to GGBN at the end of 2020. Until then all core members must have implemented the required fields in their databases.'''</div> {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- | |*RecordBasis |The specific nature of the data record, controlled vocabulary! |"PreservedSpecimen", "MaterialSample", "FossilSpecimen", "LivingSpecimen", "HumanObservation", "MachineObservation"; for Tissue or DNA samples use "MaterialSample" |- | |*materialSampleType |Classification of kind of physical sample in addition to BasisOfRecord/RecordBasis and Preparation Type |"tissue", "DNA", "specimen" |- |SpecimenUnit/Preparation |*preparationType |description of type of material, free text |or DNA: gDNA, eDNA, aDNA; for tissues/specimens: leaf, muscle, leg, blood |- |SpecimenUnit/Preparation |*preparationDate |date of DNA extraction |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparatioMaterials |extraction kit or protocoll |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparedBy |extraction staff |if unknown type "unknown" |- |SpecimenUnit/Preservation |preservation |preservation of the tissue or DNA |if unknown type "unknown" |- |SpecimenUnit/Loan/Permit |*permitStatus |Information about the presence, absence or other basic status of permits associated with the sample(s), controlled vocabulary! |Permit available, Permit not required, Permit not available, '''Unknown Material collected after 2014-10-12 cannot be in "Unknown" permit status!''' |- |SpecimenUnit/Loan/Permit |*permitStatusQualifier |Description of why a certain permit was not required or why Permit Status is unknown |"no national requirement for a permit at date of access", "officially authorized illegal holder", "collected on private land", "pre-Nagoya" |- |SpecimenUnit/Loan/Permit |*permitType |A permit is a document that allows someone to take an action that otherwise would not be allowed, controlled vocabulary! |Collecting Permit, Import Permit, Export Permit, Intellectual Property Rights, Copyright, Patent, Data use, Phytosanitary, Salvage, Exemption Permit, Material Transfer Agreement, Internationally Recognized Certificate of Compliance, Contract, Memorandum of Understanding, Memorandum of Cooperation, Veterinary Certificate, Human Pathogens, Genetically Modified Organism, Other |- |SpecimenUnit/Loan/Permit |*permitText |The text of a permit related to the gathering/shipping or further details | |- |GGBN/Amplification |amplificationDate |date of amplification; if unknown or general without content you don't have to map it |should be ISO format yyyy-mm-dd |- |GGBN/Amplification |marker | |COX1 |- |GGBN/Amplification |geneticAccessionNumber |the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD; this is a repeatable element, you can provide as much as you want |e.g. AJ45567 |- |GGBN/Amplification |genBankNumber-URI |complete link to the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD | |- |SpecimenUnit |blockedUntil |in case the sample is blocked until a specific date it's nevertheless searchable but customers cannot order it |ISO format |- |SpecimenUnit |blocked |sample is blocked (e.g. because it is consumed), but data are still available |Yes/No |- |GGBN |concentration |concentration of the DNA |65,34 |- |GGBN |@unit |unit of the concentration |µg/ml |- |GGBN |ratioOfAbsorbance260_230 |map only if filled with content |1,2 |- |GGBN |ratioOfAbsorbance260_280 |map only if filled with content |1,8 |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===The voucher identifiers=== {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Associations/UnitAssociation |*UnitID |the UnitID or CatalogueNumber used for GBIF |e.g. the barcode number of your specimens |- |Associations/UnitAssociation |*SourceInstitutionID |the SourceInstitutionID or InstitutionCode used for GBIF |e.g. the acronym of your institution |- |Associations/UnitAssociation |*SourceName |the SourceID or CollectionCode used for GBIF |e.g. the name of the collection where the specimen belongs to, e.g. "Birds" |- |Associations/UnitAssociation |*AssociationType |the Relation between the DNA and the voucher |e.g. "DNA and voucher from same individual" |- |Associations/UnitAssociation |*DatasetAccessPoint |the wrapper url of the voucher record |e.g. "http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar" |} ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} ===Multimedia items of the voucher=== These should be mapped in the specimen mapping, not the DNA mapping. {| class="wikitable" ! Category ! Element ! Discription ! Comment |- | rowspan="3" | Metadata | Title | | |- | IPR | | |- | TechnicalContact | | |- | Unit | UnitID | | |} 8a05252bf73106e24e08ccb6f37f0db660e55762 998 997 2018-09-27T10:41:32Z JoergHoletschek 3 /* Metadata */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. {| class="wikitable" ! width="120px" | Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="5" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="5" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |IPRStatements/Citation |*Text | |Droege, G. (Ed.) 2008 - (continuously updated): DNA samples of the DNA bank at the BGBM (Botanic Garden and Botanical Museum Berlin-Dahlem). |- |IPRStatements/Citation |*@language |language of citation |should be "en" as literal |- |IPRStatements/Copyright |*Text | |The copyright for any material created by the DNA bank of the BGBM is reserved. The duplication or use of information and data such as texts or images is only permitted with the indication of the source or with prior approval by the BGBM. |- |IPRStatements/Copyright |*@language |language of copyright |should be "en" as literal |- |IPRStatements/TermsOfUse |*Text | |The use of the data is allowed only for non-profit scientific use and for non-profit nature conservation purpose. The data bases or part of it may only be used or copied by the written permission from the legal owner. |- |IPRStatements/TermsOfUse |*@language |language of the terms of use |should be "en" as literal |- | |IconURI |complete url path to the logo of your institution | |- |Owner |*Address |Complete Address of the institution that owns the DNA bank samples and data | |- |Owner/Representation |*Text |name of your institution |Botanic Garden an Botanical Museum Berlin-Dahlem (Freie Universität Berlin) |- |Owner |*URL |path to website of your institution |http://www.bgbm.org |- |Unit |*RecordBasis |this value must be part of the ABCD vocabulary |must be "MaterialSample" |- |Unit |*KindOfUnit |the type of sample |e.g. DNA |} ==Sample data== ===The sample identifiers=== We use identifier the same way as GBIF does. In addition to the traditional triple ID (see below) you can also provide the GUID. '''Please note that UnitID should be unique in one database!''' Mostly the extraction number is used for it. {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Unit |*<span style="color:red">SourceID</span> |short description of relevant collection |should be "DNA Bank" or "tissue collection" |- |Unit |*<span style="color:red">SourceInstitutionID</span> |short name/abbreviation of relevant institution |BGBM |- |Unit |*<span style="color:red">UnitID</span> |DNA extraction number or tissue number |DNA 123 |- |Unit |UnitGUID |sample record GUID |f6c7fd5a-2f04-4dcb-ba38-0230bf196d30 |} ===GGBN terms=== The following fields are highly recommended, some are mandatory (marked with *), but feel free to map more! <div id="wikinote">'''Please have a look at the [http://terms.tdwg.org/wiki/GGBN_Data_Standard GGBN Data Standard] documentation where all GGBN terms are described in detail. The table below is just listing the most important ones. Since the Nagoya Protocol has come into force we higly recommend to map the Permit fields. They will become mandatory for providing data to GGBN at the end of 2020. Until then all core members must have implemented the required fields in their databases.'''</div> {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- | |*RecordBasis |The specific nature of the data record, controlled vocabulary! |"PreservedSpecimen", "MaterialSample", "FossilSpecimen", "LivingSpecimen", "HumanObservation", "MachineObservation"; for Tissue or DNA samples use "MaterialSample" |- | |*materialSampleType |Classification of kind of physical sample in addition to BasisOfRecord/RecordBasis and Preparation Type |"tissue", "DNA", "specimen" |- |SpecimenUnit/Preparation |*preparationType |description of type of material, free text |or DNA: gDNA, eDNA, aDNA; for tissues/specimens: leaf, muscle, leg, blood |- |SpecimenUnit/Preparation |*preparationDate |date of DNA extraction |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparatioMaterials |extraction kit or protocoll |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparedBy |extraction staff |if unknown type "unknown" |- |SpecimenUnit/Preservation |preservation |preservation of the tissue or DNA |if unknown type "unknown" |- |SpecimenUnit/Loan/Permit |*permitStatus |Information about the presence, absence or other basic status of permits associated with the sample(s), controlled vocabulary! |Permit available, Permit not required, Permit not available, '''Unknown Material collected after 2014-10-12 cannot be in "Unknown" permit status!''' |- |SpecimenUnit/Loan/Permit |*permitStatusQualifier |Description of why a certain permit was not required or why Permit Status is unknown |"no national requirement for a permit at date of access", "officially authorized illegal holder", "collected on private land", "pre-Nagoya" |- |SpecimenUnit/Loan/Permit |*permitType |A permit is a document that allows someone to take an action that otherwise would not be allowed, controlled vocabulary! |Collecting Permit, Import Permit, Export Permit, Intellectual Property Rights, Copyright, Patent, Data use, Phytosanitary, Salvage, Exemption Permit, Material Transfer Agreement, Internationally Recognized Certificate of Compliance, Contract, Memorandum of Understanding, Memorandum of Cooperation, Veterinary Certificate, Human Pathogens, Genetically Modified Organism, Other |- |SpecimenUnit/Loan/Permit |*permitText |The text of a permit related to the gathering/shipping or further details | |- |GGBN/Amplification |amplificationDate |date of amplification; if unknown or general without content you don't have to map it |should be ISO format yyyy-mm-dd |- |GGBN/Amplification |marker | |COX1 |- |GGBN/Amplification |geneticAccessionNumber |the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD; this is a repeatable element, you can provide as much as you want |e.g. AJ45567 |- |GGBN/Amplification |genBankNumber-URI |complete link to the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD | |- |SpecimenUnit |blockedUntil |in case the sample is blocked until a specific date it's nevertheless searchable but customers cannot order it |ISO format |- |SpecimenUnit |blocked |sample is blocked (e.g. because it is consumed), but data are still available |Yes/No |- |GGBN |concentration |concentration of the DNA |65,34 |- |GGBN |@unit |unit of the concentration |µg/ml |- |GGBN |ratioOfAbsorbance260_230 |map only if filled with content |1,2 |- |GGBN |ratioOfAbsorbance260_280 |map only if filled with content |1,8 |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===The voucher identifiers=== {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Associations/UnitAssociation |*UnitID |the UnitID or CatalogueNumber used for GBIF |e.g. the barcode number of your specimens |- |Associations/UnitAssociation |*SourceInstitutionID |the SourceInstitutionID or InstitutionCode used for GBIF |e.g. the acronym of your institution |- |Associations/UnitAssociation |*SourceName |the SourceID or CollectionCode used for GBIF |e.g. the name of the collection where the specimen belongs to, e.g. "Birds" |- |Associations/UnitAssociation |*AssociationType |the Relation between the DNA and the voucher |e.g. "DNA and voucher from same individual" |- |Associations/UnitAssociation |*DatasetAccessPoint |the wrapper url of the voucher record |e.g. "http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar" |} ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} ===Multimedia items of the voucher=== These should be mapped in the specimen mapping, not the DNA mapping. {| class="wikitable" ! Category ! Element ! Discription ! Comment |- | rowspan="3" | Metadata | Title | | |- | IPR | | |- | TechnicalContact | | |- | Unit | UnitID | | |} 330dc5dbb5750d5a593308fb9d279836ae9dcb7f 999 998 2018-09-27T10:44:10Z JoergHoletschek 3 /* Metadata */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" ! width="120px" | Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="5" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="5" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |} === IPR Statements === {| class="wikitable" ! width="120px" | Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |IPRStatements/Citation |*Text | |Droege, G. (Ed.) 2008 - (continuously updated): DNA samples of the DNA bank at the BGBM (Botanic Garden and Botanical Museum Berlin-Dahlem). |- |IPRStatements/Citation |*@language |language of citation |should be "en" as literal |- |IPRStatements/Copyright |*Text | |The copyright for any material created by the DNA bank of the BGBM is reserved. The duplication or use of information and data such as texts or images is only permitted with the indication of the source or with prior approval by the BGBM. |- |IPRStatements/Copyright |*@language |language of copyright |should be "en" as literal |- |IPRStatements/TermsOfUse |*Text | |The use of the data is allowed only for non-profit scientific use and for non-profit nature conservation purpose. The data bases or part of it may only be used or copied by the written permission from the legal owner. |- |IPRStatements/TermsOfUse |*@language |language of the terms of use |should be "en" as literal |- | |IconURI |complete url path to the logo of your institution | |- |Owner |*Address |Complete Address of the institution that owns the DNA bank samples and data | |- |Owner/Representation |*Text |name of your institution |Botanic Garden an Botanical Museum Berlin-Dahlem (Freie Universität Berlin) |- |Owner |*URL |path to website of your institution |http://www.bgbm.org |- |Unit |*RecordBasis |this value must be part of the ABCD vocabulary |must be "MaterialSample" |- |Unit |*KindOfUnit |the type of sample |e.g. DNA |} ==Sample data== ===The sample identifiers=== We use identifier the same way as GBIF does. In addition to the traditional triple ID (see below) you can also provide the GUID. '''Please note that UnitID should be unique in one database!''' Mostly the extraction number is used for it. {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Unit |*<span style="color:red">SourceID</span> |short description of relevant collection |should be "DNA Bank" or "tissue collection" |- |Unit |*<span style="color:red">SourceInstitutionID</span> |short name/abbreviation of relevant institution |BGBM |- |Unit |*<span style="color:red">UnitID</span> |DNA extraction number or tissue number |DNA 123 |- |Unit |UnitGUID |sample record GUID |f6c7fd5a-2f04-4dcb-ba38-0230bf196d30 |} ===GGBN terms=== The following fields are highly recommended, some are mandatory (marked with *), but feel free to map more! <div id="wikinote">'''Please have a look at the [http://terms.tdwg.org/wiki/GGBN_Data_Standard GGBN Data Standard] documentation where all GGBN terms are described in detail. The table below is just listing the most important ones. Since the Nagoya Protocol has come into force we higly recommend to map the Permit fields. They will become mandatory for providing data to GGBN at the end of 2020. Until then all core members must have implemented the required fields in their databases.'''</div> {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- | |*RecordBasis |The specific nature of the data record, controlled vocabulary! |"PreservedSpecimen", "MaterialSample", "FossilSpecimen", "LivingSpecimen", "HumanObservation", "MachineObservation"; for Tissue or DNA samples use "MaterialSample" |- | |*materialSampleType |Classification of kind of physical sample in addition to BasisOfRecord/RecordBasis and Preparation Type |"tissue", "DNA", "specimen" |- |SpecimenUnit/Preparation |*preparationType |description of type of material, free text |or DNA: gDNA, eDNA, aDNA; for tissues/specimens: leaf, muscle, leg, blood |- |SpecimenUnit/Preparation |*preparationDate |date of DNA extraction |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparatioMaterials |extraction kit or protocoll |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparedBy |extraction staff |if unknown type "unknown" |- |SpecimenUnit/Preservation |preservation |preservation of the tissue or DNA |if unknown type "unknown" |- |SpecimenUnit/Loan/Permit |*permitStatus |Information about the presence, absence or other basic status of permits associated with the sample(s), controlled vocabulary! |Permit available, Permit not required, Permit not available, '''Unknown Material collected after 2014-10-12 cannot be in "Unknown" permit status!''' |- |SpecimenUnit/Loan/Permit |*permitStatusQualifier |Description of why a certain permit was not required or why Permit Status is unknown |"no national requirement for a permit at date of access", "officially authorized illegal holder", "collected on private land", "pre-Nagoya" |- |SpecimenUnit/Loan/Permit |*permitType |A permit is a document that allows someone to take an action that otherwise would not be allowed, controlled vocabulary! |Collecting Permit, Import Permit, Export Permit, Intellectual Property Rights, Copyright, Patent, Data use, Phytosanitary, Salvage, Exemption Permit, Material Transfer Agreement, Internationally Recognized Certificate of Compliance, Contract, Memorandum of Understanding, Memorandum of Cooperation, Veterinary Certificate, Human Pathogens, Genetically Modified Organism, Other |- |SpecimenUnit/Loan/Permit |*permitText |The text of a permit related to the gathering/shipping or further details | |- |GGBN/Amplification |amplificationDate |date of amplification; if unknown or general without content you don't have to map it |should be ISO format yyyy-mm-dd |- |GGBN/Amplification |marker | |COX1 |- |GGBN/Amplification |geneticAccessionNumber |the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD; this is a repeatable element, you can provide as much as you want |e.g. AJ45567 |- |GGBN/Amplification |genBankNumber-URI |complete link to the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD | |- |SpecimenUnit |blockedUntil |in case the sample is blocked until a specific date it's nevertheless searchable but customers cannot order it |ISO format |- |SpecimenUnit |blocked |sample is blocked (e.g. because it is consumed), but data are still available |Yes/No |- |GGBN |concentration |concentration of the DNA |65,34 |- |GGBN |@unit |unit of the concentration |µg/ml |- |GGBN |ratioOfAbsorbance260_230 |map only if filled with content |1,2 |- |GGBN |ratioOfAbsorbance260_280 |map only if filled with content |1,8 |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===The voucher identifiers=== {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Associations/UnitAssociation |*UnitID |the UnitID or CatalogueNumber used for GBIF |e.g. the barcode number of your specimens |- |Associations/UnitAssociation |*SourceInstitutionID |the SourceInstitutionID or InstitutionCode used for GBIF |e.g. the acronym of your institution |- |Associations/UnitAssociation |*SourceName |the SourceID or CollectionCode used for GBIF |e.g. the name of the collection where the specimen belongs to, e.g. "Birds" |- |Associations/UnitAssociation |*AssociationType |the Relation between the DNA and the voucher |e.g. "DNA and voucher from same individual" |- |Associations/UnitAssociation |*DatasetAccessPoint |the wrapper url of the voucher record |e.g. "http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar" |} ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} ===Multimedia items of the voucher=== These should be mapped in the specimen mapping, not the DNA mapping. {| class="wikitable" ! Category ! Element ! Discription ! Comment |- | rowspan="3" | Metadata | Title | | |- | IPR | | |- | TechnicalContact | | |- | Unit | UnitID | | |} 978299edb4fcd6c9a3948c892d752e2033e4294f 1000 999 2018-09-27T10:50:17Z JoergHoletschek 3 /* General metadata */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" ! width="120px" | Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |-} === IPR Statements === {| class="wikitable" ! width="120px" | Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |IPRStatements/Citation |*Text | |Droege, G. (Ed.) 2008 - (continuously updated): DNA samples of the DNA bank at the BGBM (Botanic Garden and Botanical Museum Berlin-Dahlem). |- |IPRStatements/Citation |*@language |language of citation |should be "en" as literal |- |IPRStatements/Copyright |*Text | |The copyright for any material created by the DNA bank of the BGBM is reserved. The duplication or use of information and data such as texts or images is only permitted with the indication of the source or with prior approval by the BGBM. |- |IPRStatements/Copyright |*@language |language of copyright |should be "en" as literal |- |IPRStatements/TermsOfUse |*Text | |The use of the data is allowed only for non-profit scientific use and for non-profit nature conservation purpose. The data bases or part of it may only be used or copied by the written permission from the legal owner. |- |IPRStatements/TermsOfUse |*@language |language of the terms of use |should be "en" as literal |- | |IconURI |complete url path to the logo of your institution | |- |Owner |*Address |Complete Address of the institution that owns the DNA bank samples and data | |- |Owner/Representation |*Text |name of your institution |Botanic Garden an Botanical Museum Berlin-Dahlem (Freie Universität Berlin) |- |Owner |*URL |path to website of your institution |http://www.bgbm.org |- |Unit |*RecordBasis |this value must be part of the ABCD vocabulary |must be "MaterialSample" |- |Unit |*KindOfUnit |the type of sample |e.g. DNA |} ==Sample data== ===The sample identifiers=== We use identifier the same way as GBIF does. In addition to the traditional triple ID (see below) you can also provide the GUID. '''Please note that UnitID should be unique in one database!''' Mostly the extraction number is used for it. {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Unit |*<span style="color:red">SourceID</span> |short description of relevant collection |should be "DNA Bank" or "tissue collection" |- |Unit |*<span style="color:red">SourceInstitutionID</span> |short name/abbreviation of relevant institution |BGBM |- |Unit |*<span style="color:red">UnitID</span> |DNA extraction number or tissue number |DNA 123 |- |Unit |UnitGUID |sample record GUID |f6c7fd5a-2f04-4dcb-ba38-0230bf196d30 |} ===GGBN terms=== The following fields are highly recommended, some are mandatory (marked with *), but feel free to map more! <div id="wikinote">'''Please have a look at the [http://terms.tdwg.org/wiki/GGBN_Data_Standard GGBN Data Standard] documentation where all GGBN terms are described in detail. The table below is just listing the most important ones. Since the Nagoya Protocol has come into force we higly recommend to map the Permit fields. They will become mandatory for providing data to GGBN at the end of 2020. Until then all core members must have implemented the required fields in their databases.'''</div> {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- | |*RecordBasis |The specific nature of the data record, controlled vocabulary! |"PreservedSpecimen", "MaterialSample", "FossilSpecimen", "LivingSpecimen", "HumanObservation", "MachineObservation"; for Tissue or DNA samples use "MaterialSample" |- | |*materialSampleType |Classification of kind of physical sample in addition to BasisOfRecord/RecordBasis and Preparation Type |"tissue", "DNA", "specimen" |- |SpecimenUnit/Preparation |*preparationType |description of type of material, free text |or DNA: gDNA, eDNA, aDNA; for tissues/specimens: leaf, muscle, leg, blood |- |SpecimenUnit/Preparation |*preparationDate |date of DNA extraction |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparatioMaterials |extraction kit or protocoll |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparedBy |extraction staff |if unknown type "unknown" |- |SpecimenUnit/Preservation |preservation |preservation of the tissue or DNA |if unknown type "unknown" |- |SpecimenUnit/Loan/Permit |*permitStatus |Information about the presence, absence or other basic status of permits associated with the sample(s), controlled vocabulary! |Permit available, Permit not required, Permit not available, '''Unknown Material collected after 2014-10-12 cannot be in "Unknown" permit status!''' |- |SpecimenUnit/Loan/Permit |*permitStatusQualifier |Description of why a certain permit was not required or why Permit Status is unknown |"no national requirement for a permit at date of access", "officially authorized illegal holder", "collected on private land", "pre-Nagoya" |- |SpecimenUnit/Loan/Permit |*permitType |A permit is a document that allows someone to take an action that otherwise would not be allowed, controlled vocabulary! |Collecting Permit, Import Permit, Export Permit, Intellectual Property Rights, Copyright, Patent, Data use, Phytosanitary, Salvage, Exemption Permit, Material Transfer Agreement, Internationally Recognized Certificate of Compliance, Contract, Memorandum of Understanding, Memorandum of Cooperation, Veterinary Certificate, Human Pathogens, Genetically Modified Organism, Other |- |SpecimenUnit/Loan/Permit |*permitText |The text of a permit related to the gathering/shipping or further details | |- |GGBN/Amplification |amplificationDate |date of amplification; if unknown or general without content you don't have to map it |should be ISO format yyyy-mm-dd |- |GGBN/Amplification |marker | |COX1 |- |GGBN/Amplification |geneticAccessionNumber |the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD; this is a repeatable element, you can provide as much as you want |e.g. AJ45567 |- |GGBN/Amplification |genBankNumber-URI |complete link to the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD | |- |SpecimenUnit |blockedUntil |in case the sample is blocked until a specific date it's nevertheless searchable but customers cannot order it |ISO format |- |SpecimenUnit |blocked |sample is blocked (e.g. because it is consumed), but data are still available |Yes/No |- |GGBN |concentration |concentration of the DNA |65,34 |- |GGBN |@unit |unit of the concentration |µg/ml |- |GGBN |ratioOfAbsorbance260_230 |map only if filled with content |1,2 |- |GGBN |ratioOfAbsorbance260_280 |map only if filled with content |1,8 |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===The voucher identifiers=== {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Associations/UnitAssociation |*UnitID |the UnitID or CatalogueNumber used for GBIF |e.g. the barcode number of your specimens |- |Associations/UnitAssociation |*SourceInstitutionID |the SourceInstitutionID or InstitutionCode used for GBIF |e.g. the acronym of your institution |- |Associations/UnitAssociation |*SourceName |the SourceID or CollectionCode used for GBIF |e.g. the name of the collection where the specimen belongs to, e.g. "Birds" |- |Associations/UnitAssociation |*AssociationType |the Relation between the DNA and the voucher |e.g. "DNA and voucher from same individual" |- |Associations/UnitAssociation |*DatasetAccessPoint |the wrapper url of the voucher record |e.g. "http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar" |} ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} ===Multimedia items of the voucher=== These should be mapped in the specimen mapping, not the DNA mapping. {| class="wikitable" ! Category ! Element ! Discription ! Comment |- | rowspan="3" | Metadata | Title | | |- | IPR | | |- | TechnicalContact | | |- | Unit | UnitID | | |} 042d000e3533774bc410766a36cc9975a03b724c 1001 1000 2018-09-27T10:50:53Z JoergHoletschek 3 /* Metadata */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" ! width="120px" | Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === {| class="wikitable" ! width="120px" | Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |IPRStatements/Citation |*Text | |Droege, G. (Ed.) 2008 - (continuously updated): DNA samples of the DNA bank at the BGBM (Botanic Garden and Botanical Museum Berlin-Dahlem). |- |IPRStatements/Citation |*@language |language of citation |should be "en" as literal |- |IPRStatements/Copyright |*Text | |The copyright for any material created by the DNA bank of the BGBM is reserved. The duplication or use of information and data such as texts or images is only permitted with the indication of the source or with prior approval by the BGBM. |- |IPRStatements/Copyright |*@language |language of copyright |should be "en" as literal |- |IPRStatements/TermsOfUse |*Text | |The use of the data is allowed only for non-profit scientific use and for non-profit nature conservation purpose. The data bases or part of it may only be used or copied by the written permission from the legal owner. |- |IPRStatements/TermsOfUse |*@language |language of the terms of use |should be "en" as literal |- | |IconURI |complete url path to the logo of your institution | |- |Owner |*Address |Complete Address of the institution that owns the DNA bank samples and data | |- |Owner/Representation |*Text |name of your institution |Botanic Garden an Botanical Museum Berlin-Dahlem (Freie Universität Berlin) |- |Owner |*URL |path to website of your institution |http://www.bgbm.org |- |Unit |*RecordBasis |this value must be part of the ABCD vocabulary |must be "MaterialSample" |- |Unit |*KindOfUnit |the type of sample |e.g. DNA |} ==Sample data== ===The sample identifiers=== We use identifier the same way as GBIF does. In addition to the traditional triple ID (see below) you can also provide the GUID. '''Please note that UnitID should be unique in one database!''' Mostly the extraction number is used for it. {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Unit |*<span style="color:red">SourceID</span> |short description of relevant collection |should be "DNA Bank" or "tissue collection" |- |Unit |*<span style="color:red">SourceInstitutionID</span> |short name/abbreviation of relevant institution |BGBM |- |Unit |*<span style="color:red">UnitID</span> |DNA extraction number or tissue number |DNA 123 |- |Unit |UnitGUID |sample record GUID |f6c7fd5a-2f04-4dcb-ba38-0230bf196d30 |} ===GGBN terms=== The following fields are highly recommended, some are mandatory (marked with *), but feel free to map more! <div id="wikinote">'''Please have a look at the [http://terms.tdwg.org/wiki/GGBN_Data_Standard GGBN Data Standard] documentation where all GGBN terms are described in detail. The table below is just listing the most important ones. Since the Nagoya Protocol has come into force we higly recommend to map the Permit fields. They will become mandatory for providing data to GGBN at the end of 2020. Until then all core members must have implemented the required fields in their databases.'''</div> {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- | |*RecordBasis |The specific nature of the data record, controlled vocabulary! |"PreservedSpecimen", "MaterialSample", "FossilSpecimen", "LivingSpecimen", "HumanObservation", "MachineObservation"; for Tissue or DNA samples use "MaterialSample" |- | |*materialSampleType |Classification of kind of physical sample in addition to BasisOfRecord/RecordBasis and Preparation Type |"tissue", "DNA", "specimen" |- |SpecimenUnit/Preparation |*preparationType |description of type of material, free text |or DNA: gDNA, eDNA, aDNA; for tissues/specimens: leaf, muscle, leg, blood |- |SpecimenUnit/Preparation |*preparationDate |date of DNA extraction |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparatioMaterials |extraction kit or protocoll |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparedBy |extraction staff |if unknown type "unknown" |- |SpecimenUnit/Preservation |preservation |preservation of the tissue or DNA |if unknown type "unknown" |- |SpecimenUnit/Loan/Permit |*permitStatus |Information about the presence, absence or other basic status of permits associated with the sample(s), controlled vocabulary! |Permit available, Permit not required, Permit not available, '''Unknown Material collected after 2014-10-12 cannot be in "Unknown" permit status!''' |- |SpecimenUnit/Loan/Permit |*permitStatusQualifier |Description of why a certain permit was not required or why Permit Status is unknown |"no national requirement for a permit at date of access", "officially authorized illegal holder", "collected on private land", "pre-Nagoya" |- |SpecimenUnit/Loan/Permit |*permitType |A permit is a document that allows someone to take an action that otherwise would not be allowed, controlled vocabulary! |Collecting Permit, Import Permit, Export Permit, Intellectual Property Rights, Copyright, Patent, Data use, Phytosanitary, Salvage, Exemption Permit, Material Transfer Agreement, Internationally Recognized Certificate of Compliance, Contract, Memorandum of Understanding, Memorandum of Cooperation, Veterinary Certificate, Human Pathogens, Genetically Modified Organism, Other |- |SpecimenUnit/Loan/Permit |*permitText |The text of a permit related to the gathering/shipping or further details | |- |GGBN/Amplification |amplificationDate |date of amplification; if unknown or general without content you don't have to map it |should be ISO format yyyy-mm-dd |- |GGBN/Amplification |marker | |COX1 |- |GGBN/Amplification |geneticAccessionNumber |the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD; this is a repeatable element, you can provide as much as you want |e.g. AJ45567 |- |GGBN/Amplification |genBankNumber-URI |complete link to the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD | |- |SpecimenUnit |blockedUntil |in case the sample is blocked until a specific date it's nevertheless searchable but customers cannot order it |ISO format |- |SpecimenUnit |blocked |sample is blocked (e.g. because it is consumed), but data are still available |Yes/No |- |GGBN |concentration |concentration of the DNA |65,34 |- |GGBN |@unit |unit of the concentration |µg/ml |- |GGBN |ratioOfAbsorbance260_230 |map only if filled with content |1,2 |- |GGBN |ratioOfAbsorbance260_280 |map only if filled with content |1,8 |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===The voucher identifiers=== {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Associations/UnitAssociation |*UnitID |the UnitID or CatalogueNumber used for GBIF |e.g. the barcode number of your specimens |- |Associations/UnitAssociation |*SourceInstitutionID |the SourceInstitutionID or InstitutionCode used for GBIF |e.g. the acronym of your institution |- |Associations/UnitAssociation |*SourceName |the SourceID or CollectionCode used for GBIF |e.g. the name of the collection where the specimen belongs to, e.g. "Birds" |- |Associations/UnitAssociation |*AssociationType |the Relation between the DNA and the voucher |e.g. "DNA and voucher from same individual" |- |Associations/UnitAssociation |*DatasetAccessPoint |the wrapper url of the voucher record |e.g. "http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar" |} ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} ===Multimedia items of the voucher=== These should be mapped in the specimen mapping, not the DNA mapping. {| class="wikitable" ! Category ! Element ! Discription ! Comment |- | rowspan="3" | Metadata | Title | | |- | IPR | | |- | TechnicalContact | | |- | Unit | UnitID | | |} 03b7c416b002f9f6ed9aa6df45da61cc84f25674 1002 1001 2018-09-27T10:56:42Z JoergHoletschek 3 /* General metadata */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" ! width="120px" | Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === {| class="wikitable" ! width="120px" | Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |IPRStatements/Citation |*Text | |Droege, G. (Ed.) 2008 - (continuously updated): DNA samples of the DNA bank at the BGBM (Botanic Garden and Botanical Museum Berlin-Dahlem). |- |IPRStatements/Citation |*@language |language of citation |should be "en" as literal |- |IPRStatements/Copyright |*Text | |The copyright for any material created by the DNA bank of the BGBM is reserved. The duplication or use of information and data such as texts or images is only permitted with the indication of the source or with prior approval by the BGBM. |- |IPRStatements/Copyright |*@language |language of copyright |should be "en" as literal |- |IPRStatements/TermsOfUse |*Text | |The use of the data is allowed only for non-profit scientific use and for non-profit nature conservation purpose. The data bases or part of it may only be used or copied by the written permission from the legal owner. |- |IPRStatements/TermsOfUse |*@language |language of the terms of use |should be "en" as literal |- | |IconURI |complete url path to the logo of your institution | |- |Owner |*Address |Complete Address of the institution that owns the DNA bank samples and data | |- |Owner/Representation |*Text |name of your institution |Botanic Garden an Botanical Museum Berlin-Dahlem (Freie Universität Berlin) |- |Owner |*URL |path to website of your institution |http://www.bgbm.org |- |Unit |*RecordBasis |this value must be part of the ABCD vocabulary |must be "MaterialSample" |- |Unit |*KindOfUnit |the type of sample |e.g. DNA |} ==Sample data== ===The sample identifiers=== We use identifier the same way as GBIF does. In addition to the traditional triple ID (see below) you can also provide the GUID. '''Please note that UnitID should be unique in one database!''' Mostly the extraction number is used for it. {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Unit |*<span style="color:red">SourceID</span> |short description of relevant collection |should be "DNA Bank" or "tissue collection" |- |Unit |*<span style="color:red">SourceInstitutionID</span> |short name/abbreviation of relevant institution |BGBM |- |Unit |*<span style="color:red">UnitID</span> |DNA extraction number or tissue number |DNA 123 |- |Unit |UnitGUID |sample record GUID |f6c7fd5a-2f04-4dcb-ba38-0230bf196d30 |} ===GGBN terms=== The following fields are highly recommended, some are mandatory (marked with *), but feel free to map more! <div id="wikinote">'''Please have a look at the [http://terms.tdwg.org/wiki/GGBN_Data_Standard GGBN Data Standard] documentation where all GGBN terms are described in detail. The table below is just listing the most important ones. Since the Nagoya Protocol has come into force we higly recommend to map the Permit fields. They will become mandatory for providing data to GGBN at the end of 2020. Until then all core members must have implemented the required fields in their databases.'''</div> {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- | |*RecordBasis |The specific nature of the data record, controlled vocabulary! |"PreservedSpecimen", "MaterialSample", "FossilSpecimen", "LivingSpecimen", "HumanObservation", "MachineObservation"; for Tissue or DNA samples use "MaterialSample" |- | |*materialSampleType |Classification of kind of physical sample in addition to BasisOfRecord/RecordBasis and Preparation Type |"tissue", "DNA", "specimen" |- |SpecimenUnit/Preparation |*preparationType |description of type of material, free text |or DNA: gDNA, eDNA, aDNA; for tissues/specimens: leaf, muscle, leg, blood |- |SpecimenUnit/Preparation |*preparationDate |date of DNA extraction |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparatioMaterials |extraction kit or protocoll |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparedBy |extraction staff |if unknown type "unknown" |- |SpecimenUnit/Preservation |preservation |preservation of the tissue or DNA |if unknown type "unknown" |- |SpecimenUnit/Loan/Permit |*permitStatus |Information about the presence, absence or other basic status of permits associated with the sample(s), controlled vocabulary! |Permit available, Permit not required, Permit not available, '''Unknown Material collected after 2014-10-12 cannot be in "Unknown" permit status!''' |- |SpecimenUnit/Loan/Permit |*permitStatusQualifier |Description of why a certain permit was not required or why Permit Status is unknown |"no national requirement for a permit at date of access", "officially authorized illegal holder", "collected on private land", "pre-Nagoya" |- |SpecimenUnit/Loan/Permit |*permitType |A permit is a document that allows someone to take an action that otherwise would not be allowed, controlled vocabulary! |Collecting Permit, Import Permit, Export Permit, Intellectual Property Rights, Copyright, Patent, Data use, Phytosanitary, Salvage, Exemption Permit, Material Transfer Agreement, Internationally Recognized Certificate of Compliance, Contract, Memorandum of Understanding, Memorandum of Cooperation, Veterinary Certificate, Human Pathogens, Genetically Modified Organism, Other |- |SpecimenUnit/Loan/Permit |*permitText |The text of a permit related to the gathering/shipping or further details | |- |GGBN/Amplification |amplificationDate |date of amplification; if unknown or general without content you don't have to map it |should be ISO format yyyy-mm-dd |- |GGBN/Amplification |marker | |COX1 |- |GGBN/Amplification |geneticAccessionNumber |the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD; this is a repeatable element, you can provide as much as you want |e.g. AJ45567 |- |GGBN/Amplification |genBankNumber-URI |complete link to the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD | |- |SpecimenUnit |blockedUntil |in case the sample is blocked until a specific date it's nevertheless searchable but customers cannot order it |ISO format |- |SpecimenUnit |blocked |sample is blocked (e.g. because it is consumed), but data are still available |Yes/No |- |GGBN |concentration |concentration of the DNA |65,34 |- |GGBN |@unit |unit of the concentration |µg/ml |- |GGBN |ratioOfAbsorbance260_230 |map only if filled with content |1,2 |- |GGBN |ratioOfAbsorbance260_280 |map only if filled with content |1,8 |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===The voucher identifiers=== {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Associations/UnitAssociation |*UnitID |the UnitID or CatalogueNumber used for GBIF |e.g. the barcode number of your specimens |- |Associations/UnitAssociation |*SourceInstitutionID |the SourceInstitutionID or InstitutionCode used for GBIF |e.g. the acronym of your institution |- |Associations/UnitAssociation |*SourceName |the SourceID or CollectionCode used for GBIF |e.g. the name of the collection where the specimen belongs to, e.g. "Birds" |- |Associations/UnitAssociation |*AssociationType |the Relation between the DNA and the voucher |e.g. "DNA and voucher from same individual" |- |Associations/UnitAssociation |*DatasetAccessPoint |the wrapper url of the voucher record |e.g. "http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar" |} ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} ===Multimedia items of the voucher=== These should be mapped in the specimen mapping, not the DNA mapping. {| class="wikitable" ! Category ! Element ! Discription ! Comment |- | rowspan="3" | Metadata | Title | | |- | IPR | | |- | TechnicalContact | | |- | Unit | UnitID | | |} 7aad6cf951a8d7402ad4cb43acdb3fe38e07c569 1003 1002 2018-09-27T11:40:01Z JoergHoletschek 3 /* IPR Statements */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" ! width="120px" | Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains seven sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements | | |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer | | |- |Copyright | | |- |IPRDeclaration | | |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Sample data== ===The sample identifiers=== We use identifier the same way as GBIF does. In addition to the traditional triple ID (see below) you can also provide the GUID. '''Please note that UnitID should be unique in one database!''' Mostly the extraction number is used for it. {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Unit |*<span style="color:red">SourceID</span> |short description of relevant collection |should be "DNA Bank" or "tissue collection" |- |Unit |*<span style="color:red">SourceInstitutionID</span> |short name/abbreviation of relevant institution |BGBM |- |Unit |*<span style="color:red">UnitID</span> |DNA extraction number or tissue number |DNA 123 |- |Unit |UnitGUID |sample record GUID |f6c7fd5a-2f04-4dcb-ba38-0230bf196d30 |} ===GGBN terms=== The following fields are highly recommended, some are mandatory (marked with *), but feel free to map more! <div id="wikinote">'''Please have a look at the [http://terms.tdwg.org/wiki/GGBN_Data_Standard GGBN Data Standard] documentation where all GGBN terms are described in detail. The table below is just listing the most important ones. Since the Nagoya Protocol has come into force we higly recommend to map the Permit fields. They will become mandatory for providing data to GGBN at the end of 2020. Until then all core members must have implemented the required fields in their databases.'''</div> {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- | |*RecordBasis |The specific nature of the data record, controlled vocabulary! |"PreservedSpecimen", "MaterialSample", "FossilSpecimen", "LivingSpecimen", "HumanObservation", "MachineObservation"; for Tissue or DNA samples use "MaterialSample" |- | |*materialSampleType |Classification of kind of physical sample in addition to BasisOfRecord/RecordBasis and Preparation Type |"tissue", "DNA", "specimen" |- |SpecimenUnit/Preparation |*preparationType |description of type of material, free text |or DNA: gDNA, eDNA, aDNA; for tissues/specimens: leaf, muscle, leg, blood |- |SpecimenUnit/Preparation |*preparationDate |date of DNA extraction |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparatioMaterials |extraction kit or protocoll |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparedBy |extraction staff |if unknown type "unknown" |- |SpecimenUnit/Preservation |preservation |preservation of the tissue or DNA |if unknown type "unknown" |- |SpecimenUnit/Loan/Permit |*permitStatus |Information about the presence, absence or other basic status of permits associated with the sample(s), controlled vocabulary! |Permit available, Permit not required, Permit not available, '''Unknown Material collected after 2014-10-12 cannot be in "Unknown" permit status!''' |- |SpecimenUnit/Loan/Permit |*permitStatusQualifier |Description of why a certain permit was not required or why Permit Status is unknown |"no national requirement for a permit at date of access", "officially authorized illegal holder", "collected on private land", "pre-Nagoya" |- |SpecimenUnit/Loan/Permit |*permitType |A permit is a document that allows someone to take an action that otherwise would not be allowed, controlled vocabulary! |Collecting Permit, Import Permit, Export Permit, Intellectual Property Rights, Copyright, Patent, Data use, Phytosanitary, Salvage, Exemption Permit, Material Transfer Agreement, Internationally Recognized Certificate of Compliance, Contract, Memorandum of Understanding, Memorandum of Cooperation, Veterinary Certificate, Human Pathogens, Genetically Modified Organism, Other |- |SpecimenUnit/Loan/Permit |*permitText |The text of a permit related to the gathering/shipping or further details | |- |GGBN/Amplification |amplificationDate |date of amplification; if unknown or general without content you don't have to map it |should be ISO format yyyy-mm-dd |- |GGBN/Amplification |marker | |COX1 |- |GGBN/Amplification |geneticAccessionNumber |the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD; this is a repeatable element, you can provide as much as you want |e.g. AJ45567 |- |GGBN/Amplification |genBankNumber-URI |complete link to the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD | |- |SpecimenUnit |blockedUntil |in case the sample is blocked until a specific date it's nevertheless searchable but customers cannot order it |ISO format |- |SpecimenUnit |blocked |sample is blocked (e.g. because it is consumed), but data are still available |Yes/No |- |GGBN |concentration |concentration of the DNA |65,34 |- |GGBN |@unit |unit of the concentration |µg/ml |- |GGBN |ratioOfAbsorbance260_230 |map only if filled with content |1,2 |- |GGBN |ratioOfAbsorbance260_280 |map only if filled with content |1,8 |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===The voucher identifiers=== {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Associations/UnitAssociation |*UnitID |the UnitID or CatalogueNumber used for GBIF |e.g. the barcode number of your specimens |- |Associations/UnitAssociation |*SourceInstitutionID |the SourceInstitutionID or InstitutionCode used for GBIF |e.g. the acronym of your institution |- |Associations/UnitAssociation |*SourceName |the SourceID or CollectionCode used for GBIF |e.g. the name of the collection where the specimen belongs to, e.g. "Birds" |- |Associations/UnitAssociation |*AssociationType |the Relation between the DNA and the voucher |e.g. "DNA and voucher from same individual" |- |Associations/UnitAssociation |*DatasetAccessPoint |the wrapper url of the voucher record |e.g. "http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar" |} ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} ===Multimedia items of the voucher=== These should be mapped in the specimen mapping, not the DNA mapping. {| class="wikitable" ! Category ! Element ! Discription ! Comment |- | rowspan="3" | Metadata | Title | | |- | IPR | | |- | TechnicalContact | | |- | Unit | UnitID | | |} e451175f3f0d4f18d0f5adaf056cd75e473e1312 1004 1003 2018-09-27T11:42:36Z JoergHoletschek 3 /* IPR Statements */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" ! width="120px" | Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains seven sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) | |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. | |- |Copyright | | |- |IPRDeclaration | | |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Sample data== ===The sample identifiers=== We use identifier the same way as GBIF does. In addition to the traditional triple ID (see below) you can also provide the GUID. '''Please note that UnitID should be unique in one database!''' Mostly the extraction number is used for it. {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Unit |*<span style="color:red">SourceID</span> |short description of relevant collection |should be "DNA Bank" or "tissue collection" |- |Unit |*<span style="color:red">SourceInstitutionID</span> |short name/abbreviation of relevant institution |BGBM |- |Unit |*<span style="color:red">UnitID</span> |DNA extraction number or tissue number |DNA 123 |- |Unit |UnitGUID |sample record GUID |f6c7fd5a-2f04-4dcb-ba38-0230bf196d30 |} ===GGBN terms=== The following fields are highly recommended, some are mandatory (marked with *), but feel free to map more! <div id="wikinote">'''Please have a look at the [http://terms.tdwg.org/wiki/GGBN_Data_Standard GGBN Data Standard] documentation where all GGBN terms are described in detail. The table below is just listing the most important ones. Since the Nagoya Protocol has come into force we higly recommend to map the Permit fields. They will become mandatory for providing data to GGBN at the end of 2020. Until then all core members must have implemented the required fields in their databases.'''</div> {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- | |*RecordBasis |The specific nature of the data record, controlled vocabulary! |"PreservedSpecimen", "MaterialSample", "FossilSpecimen", "LivingSpecimen", "HumanObservation", "MachineObservation"; for Tissue or DNA samples use "MaterialSample" |- | |*materialSampleType |Classification of kind of physical sample in addition to BasisOfRecord/RecordBasis and Preparation Type |"tissue", "DNA", "specimen" |- |SpecimenUnit/Preparation |*preparationType |description of type of material, free text |or DNA: gDNA, eDNA, aDNA; for tissues/specimens: leaf, muscle, leg, blood |- |SpecimenUnit/Preparation |*preparationDate |date of DNA extraction |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparatioMaterials |extraction kit or protocoll |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparedBy |extraction staff |if unknown type "unknown" |- |SpecimenUnit/Preservation |preservation |preservation of the tissue or DNA |if unknown type "unknown" |- |SpecimenUnit/Loan/Permit |*permitStatus |Information about the presence, absence or other basic status of permits associated with the sample(s), controlled vocabulary! |Permit available, Permit not required, Permit not available, '''Unknown Material collected after 2014-10-12 cannot be in "Unknown" permit status!''' |- |SpecimenUnit/Loan/Permit |*permitStatusQualifier |Description of why a certain permit was not required or why Permit Status is unknown |"no national requirement for a permit at date of access", "officially authorized illegal holder", "collected on private land", "pre-Nagoya" |- |SpecimenUnit/Loan/Permit |*permitType |A permit is a document that allows someone to take an action that otherwise would not be allowed, controlled vocabulary! |Collecting Permit, Import Permit, Export Permit, Intellectual Property Rights, Copyright, Patent, Data use, Phytosanitary, Salvage, Exemption Permit, Material Transfer Agreement, Internationally Recognized Certificate of Compliance, Contract, Memorandum of Understanding, Memorandum of Cooperation, Veterinary Certificate, Human Pathogens, Genetically Modified Organism, Other |- |SpecimenUnit/Loan/Permit |*permitText |The text of a permit related to the gathering/shipping or further details | |- |GGBN/Amplification |amplificationDate |date of amplification; if unknown or general without content you don't have to map it |should be ISO format yyyy-mm-dd |- |GGBN/Amplification |marker | |COX1 |- |GGBN/Amplification |geneticAccessionNumber |the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD; this is a repeatable element, you can provide as much as you want |e.g. AJ45567 |- |GGBN/Amplification |genBankNumber-URI |complete link to the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD | |- |SpecimenUnit |blockedUntil |in case the sample is blocked until a specific date it's nevertheless searchable but customers cannot order it |ISO format |- |SpecimenUnit |blocked |sample is blocked (e.g. because it is consumed), but data are still available |Yes/No |- |GGBN |concentration |concentration of the DNA |65,34 |- |GGBN |@unit |unit of the concentration |µg/ml |- |GGBN |ratioOfAbsorbance260_230 |map only if filled with content |1,2 |- |GGBN |ratioOfAbsorbance260_280 |map only if filled with content |1,8 |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===The voucher identifiers=== {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Associations/UnitAssociation |*UnitID |the UnitID or CatalogueNumber used for GBIF |e.g. the barcode number of your specimens |- |Associations/UnitAssociation |*SourceInstitutionID |the SourceInstitutionID or InstitutionCode used for GBIF |e.g. the acronym of your institution |- |Associations/UnitAssociation |*SourceName |the SourceID or CollectionCode used for GBIF |e.g. the name of the collection where the specimen belongs to, e.g. "Birds" |- |Associations/UnitAssociation |*AssociationType |the Relation between the DNA and the voucher |e.g. "DNA and voucher from same individual" |- |Associations/UnitAssociation |*DatasetAccessPoint |the wrapper url of the voucher record |e.g. "http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar" |} ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} ===Multimedia items of the voucher=== These should be mapped in the specimen mapping, not the DNA mapping. {| class="wikitable" ! Category ! Element ! Discription ! Comment |- | rowspan="3" | Metadata | Title | | |- | IPR | | |- | TechnicalContact | | |- | Unit | UnitID | | |} 4794238754a43359846ea7f8744eab884fb0ff86 1005 1004 2018-09-27T11:44:38Z JoergHoletschek 3 /* IPR Statements */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" ! width="120px" | Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) | |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. | |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Sample data== ===The sample identifiers=== We use identifier the same way as GBIF does. In addition to the traditional triple ID (see below) you can also provide the GUID. '''Please note that UnitID should be unique in one database!''' Mostly the extraction number is used for it. {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Unit |*<span style="color:red">SourceID</span> |short description of relevant collection |should be "DNA Bank" or "tissue collection" |- |Unit |*<span style="color:red">SourceInstitutionID</span> |short name/abbreviation of relevant institution |BGBM |- |Unit |*<span style="color:red">UnitID</span> |DNA extraction number or tissue number |DNA 123 |- |Unit |UnitGUID |sample record GUID |f6c7fd5a-2f04-4dcb-ba38-0230bf196d30 |} ===GGBN terms=== The following fields are highly recommended, some are mandatory (marked with *), but feel free to map more! <div id="wikinote">'''Please have a look at the [http://terms.tdwg.org/wiki/GGBN_Data_Standard GGBN Data Standard] documentation where all GGBN terms are described in detail. The table below is just listing the most important ones. Since the Nagoya Protocol has come into force we higly recommend to map the Permit fields. They will become mandatory for providing data to GGBN at the end of 2020. Until then all core members must have implemented the required fields in their databases.'''</div> {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- | |*RecordBasis |The specific nature of the data record, controlled vocabulary! |"PreservedSpecimen", "MaterialSample", "FossilSpecimen", "LivingSpecimen", "HumanObservation", "MachineObservation"; for Tissue or DNA samples use "MaterialSample" |- | |*materialSampleType |Classification of kind of physical sample in addition to BasisOfRecord/RecordBasis and Preparation Type |"tissue", "DNA", "specimen" |- |SpecimenUnit/Preparation |*preparationType |description of type of material, free text |or DNA: gDNA, eDNA, aDNA; for tissues/specimens: leaf, muscle, leg, blood |- |SpecimenUnit/Preparation |*preparationDate |date of DNA extraction |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparatioMaterials |extraction kit or protocoll |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparedBy |extraction staff |if unknown type "unknown" |- |SpecimenUnit/Preservation |preservation |preservation of the tissue or DNA |if unknown type "unknown" |- |SpecimenUnit/Loan/Permit |*permitStatus |Information about the presence, absence or other basic status of permits associated with the sample(s), controlled vocabulary! |Permit available, Permit not required, Permit not available, '''Unknown Material collected after 2014-10-12 cannot be in "Unknown" permit status!''' |- |SpecimenUnit/Loan/Permit |*permitStatusQualifier |Description of why a certain permit was not required or why Permit Status is unknown |"no national requirement for a permit at date of access", "officially authorized illegal holder", "collected on private land", "pre-Nagoya" |- |SpecimenUnit/Loan/Permit |*permitType |A permit is a document that allows someone to take an action that otherwise would not be allowed, controlled vocabulary! |Collecting Permit, Import Permit, Export Permit, Intellectual Property Rights, Copyright, Patent, Data use, Phytosanitary, Salvage, Exemption Permit, Material Transfer Agreement, Internationally Recognized Certificate of Compliance, Contract, Memorandum of Understanding, Memorandum of Cooperation, Veterinary Certificate, Human Pathogens, Genetically Modified Organism, Other |- |SpecimenUnit/Loan/Permit |*permitText |The text of a permit related to the gathering/shipping or further details | |- |GGBN/Amplification |amplificationDate |date of amplification; if unknown or general without content you don't have to map it |should be ISO format yyyy-mm-dd |- |GGBN/Amplification |marker | |COX1 |- |GGBN/Amplification |geneticAccessionNumber |the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD; this is a repeatable element, you can provide as much as you want |e.g. AJ45567 |- |GGBN/Amplification |genBankNumber-URI |complete link to the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD | |- |SpecimenUnit |blockedUntil |in case the sample is blocked until a specific date it's nevertheless searchable but customers cannot order it |ISO format |- |SpecimenUnit |blocked |sample is blocked (e.g. because it is consumed), but data are still available |Yes/No |- |GGBN |concentration |concentration of the DNA |65,34 |- |GGBN |@unit |unit of the concentration |µg/ml |- |GGBN |ratioOfAbsorbance260_230 |map only if filled with content |1,2 |- |GGBN |ratioOfAbsorbance260_280 |map only if filled with content |1,8 |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===The voucher identifiers=== {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Associations/UnitAssociation |*UnitID |the UnitID or CatalogueNumber used for GBIF |e.g. the barcode number of your specimens |- |Associations/UnitAssociation |*SourceInstitutionID |the SourceInstitutionID or InstitutionCode used for GBIF |e.g. the acronym of your institution |- |Associations/UnitAssociation |*SourceName |the SourceID or CollectionCode used for GBIF |e.g. the name of the collection where the specimen belongs to, e.g. "Birds" |- |Associations/UnitAssociation |*AssociationType |the Relation between the DNA and the voucher |e.g. "DNA and voucher from same individual" |- |Associations/UnitAssociation |*DatasetAccessPoint |the wrapper url of the voucher record |e.g. "http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar" |} ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} ===Multimedia items of the voucher=== These should be mapped in the specimen mapping, not the DNA mapping. {| class="wikitable" ! Category ! Element ! Discription ! Comment |- | rowspan="3" | Metadata | Title | | |- | IPR | | |- | TechnicalContact | | |- | Unit | UnitID | | |} 06355a318b544e2e2dc64e2acabe967531220dcb 1006 1005 2018-09-27T13:45:33Z JoergHoletschek 3 wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" ! width="120px" | Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) | |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. | |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''. For each specimen, one ''Units/Unit'' will be created. ===The sample identifiers=== We use identifier the same way as GBIF does. In addition to the traditional triple ID (see below) you can also provide the GUID. '''Please note that UnitID should be unique in one database!''' Mostly the extraction number is used for it. {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Unit |*<span style="color:red">SourceID</span> |short description of relevant collection |should be "DNA Bank" or "tissue collection" |- |Unit |*<span style="color:red">SourceInstitutionID</span> |short name/abbreviation of relevant institution |BGBM |- |Unit |*<span style="color:red">UnitID</span> |DNA extraction number or tissue number |DNA 123 |- |Unit |UnitGUID |sample record GUID |f6c7fd5a-2f04-4dcb-ba38-0230bf196d30 |} ===GGBN terms=== The following fields are highly recommended, some are mandatory (marked with *), but feel free to map more! <div id="wikinote">'''Please have a look at the [http://terms.tdwg.org/wiki/GGBN_Data_Standard GGBN Data Standard] documentation where all GGBN terms are described in detail. The table below is just listing the most important ones. Since the Nagoya Protocol has come into force we higly recommend to map the Permit fields. They will become mandatory for providing data to GGBN at the end of 2020. Until then all core members must have implemented the required fields in their databases.'''</div> {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- | |*RecordBasis |The specific nature of the data record, controlled vocabulary! |"PreservedSpecimen", "MaterialSample", "FossilSpecimen", "LivingSpecimen", "HumanObservation", "MachineObservation"; for Tissue or DNA samples use "MaterialSample" |- | |*materialSampleType |Classification of kind of physical sample in addition to BasisOfRecord/RecordBasis and Preparation Type |"tissue", "DNA", "specimen" |- |SpecimenUnit/Preparation |*preparationType |description of type of material, free text |or DNA: gDNA, eDNA, aDNA; for tissues/specimens: leaf, muscle, leg, blood |- |SpecimenUnit/Preparation |*preparationDate |date of DNA extraction |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparatioMaterials |extraction kit or protocoll |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparedBy |extraction staff |if unknown type "unknown" |- |SpecimenUnit/Preservation |preservation |preservation of the tissue or DNA |if unknown type "unknown" |- |SpecimenUnit/Loan/Permit |*permitStatus |Information about the presence, absence or other basic status of permits associated with the sample(s), controlled vocabulary! |Permit available, Permit not required, Permit not available, '''Unknown Material collected after 2014-10-12 cannot be in "Unknown" permit status!''' |- |SpecimenUnit/Loan/Permit |*permitStatusQualifier |Description of why a certain permit was not required or why Permit Status is unknown |"no national requirement for a permit at date of access", "officially authorized illegal holder", "collected on private land", "pre-Nagoya" |- |SpecimenUnit/Loan/Permit |*permitType |A permit is a document that allows someone to take an action that otherwise would not be allowed, controlled vocabulary! |Collecting Permit, Import Permit, Export Permit, Intellectual Property Rights, Copyright, Patent, Data use, Phytosanitary, Salvage, Exemption Permit, Material Transfer Agreement, Internationally Recognized Certificate of Compliance, Contract, Memorandum of Understanding, Memorandum of Cooperation, Veterinary Certificate, Human Pathogens, Genetically Modified Organism, Other |- |SpecimenUnit/Loan/Permit |*permitText |The text of a permit related to the gathering/shipping or further details | |- |GGBN/Amplification |amplificationDate |date of amplification; if unknown or general without content you don't have to map it |should be ISO format yyyy-mm-dd |- |GGBN/Amplification |marker | |COX1 |- |GGBN/Amplification |geneticAccessionNumber |the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD; this is a repeatable element, you can provide as much as you want |e.g. AJ45567 |- |GGBN/Amplification |genBankNumber-URI |complete link to the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD | |- |SpecimenUnit |blockedUntil |in case the sample is blocked until a specific date it's nevertheless searchable but customers cannot order it |ISO format |- |SpecimenUnit |blocked |sample is blocked (e.g. because it is consumed), but data are still available |Yes/No |- |GGBN |concentration |concentration of the DNA |65,34 |- |GGBN |@unit |unit of the concentration |µg/ml |- |GGBN |ratioOfAbsorbance260_230 |map only if filled with content |1,2 |- |GGBN |ratioOfAbsorbance260_280 |map only if filled with content |1,8 |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===The voucher identifiers=== {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Associations/UnitAssociation |*UnitID |the UnitID or CatalogueNumber used for GBIF |e.g. the barcode number of your specimens |- |Associations/UnitAssociation |*SourceInstitutionID |the SourceInstitutionID or InstitutionCode used for GBIF |e.g. the acronym of your institution |- |Associations/UnitAssociation |*SourceName |the SourceID or CollectionCode used for GBIF |e.g. the name of the collection where the specimen belongs to, e.g. "Birds" |- |Associations/UnitAssociation |*AssociationType |the Relation between the DNA and the voucher |e.g. "DNA and voucher from same individual" |- |Associations/UnitAssociation |*DatasetAccessPoint |the wrapper url of the voucher record |e.g. "http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar" |} ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} ===Multimedia items of the voucher=== These should be mapped in the specimen mapping, not the DNA mapping. {| class="wikitable" ! Category ! Element ! Discription ! Comment |- | rowspan="3" | Metadata | Title | | |- | IPR | | |- | TechnicalContact | | |- | Unit | UnitID | | |} 3478fdef85a7611257a416cd63db1f67859f12e9 1007 1006 2018-09-27T13:53:59Z JoergHoletschek 3 /* Specimen Data */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" ! width="120px" | Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) | |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. | |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* | | |- |SourceInstitutionID* | | |- |UnitID* | | |- |UnitGUID | | |- |CollectorsFieldNumber | | |- |RecordBasis | | |- |KindOfUnit | | |- |Sex | | |- |RecordURI | | |- |Notes | | |- |DateLastEdited | | |- |LastEditor | | |- |} ===The sample identifiers=== We use identifier the same way as GBIF does. In addition to the traditional triple ID (see below) you can also provide the GUID. '''Please note that UnitID should be unique in one database!''' Mostly the extraction number is used for it. {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Unit |*<span style="color:red">SourceID</span> |short description of relevant collection |should be "DNA Bank" or "tissue collection" |- |Unit |*<span style="color:red">SourceInstitutionID</span> |short name/abbreviation of relevant institution |BGBM |- |Unit |*<span style="color:red">UnitID</span> |DNA extraction number or tissue number |DNA 123 |- |Unit |UnitGUID |sample record GUID |f6c7fd5a-2f04-4dcb-ba38-0230bf196d30 |} ===GGBN terms=== The following fields are highly recommended, some are mandatory (marked with *), but feel free to map more! <div id="wikinote">'''Please have a look at the [http://terms.tdwg.org/wiki/GGBN_Data_Standard GGBN Data Standard] documentation where all GGBN terms are described in detail. The table below is just listing the most important ones. Since the Nagoya Protocol has come into force we higly recommend to map the Permit fields. They will become mandatory for providing data to GGBN at the end of 2020. Until then all core members must have implemented the required fields in their databases.'''</div> {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- | |*RecordBasis |The specific nature of the data record, controlled vocabulary! |"PreservedSpecimen", "MaterialSample", "FossilSpecimen", "LivingSpecimen", "HumanObservation", "MachineObservation"; for Tissue or DNA samples use "MaterialSample" |- | |*materialSampleType |Classification of kind of physical sample in addition to BasisOfRecord/RecordBasis and Preparation Type |"tissue", "DNA", "specimen" |- |SpecimenUnit/Preparation |*preparationType |description of type of material, free text |or DNA: gDNA, eDNA, aDNA; for tissues/specimens: leaf, muscle, leg, blood |- |SpecimenUnit/Preparation |*preparationDate |date of DNA extraction |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparatioMaterials |extraction kit or protocoll |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparedBy |extraction staff |if unknown type "unknown" |- |SpecimenUnit/Preservation |preservation |preservation of the tissue or DNA |if unknown type "unknown" |- |SpecimenUnit/Loan/Permit |*permitStatus |Information about the presence, absence or other basic status of permits associated with the sample(s), controlled vocabulary! |Permit available, Permit not required, Permit not available, '''Unknown Material collected after 2014-10-12 cannot be in "Unknown" permit status!''' |- |SpecimenUnit/Loan/Permit |*permitStatusQualifier |Description of why a certain permit was not required or why Permit Status is unknown |"no national requirement for a permit at date of access", "officially authorized illegal holder", "collected on private land", "pre-Nagoya" |- |SpecimenUnit/Loan/Permit |*permitType |A permit is a document that allows someone to take an action that otherwise would not be allowed, controlled vocabulary! |Collecting Permit, Import Permit, Export Permit, Intellectual Property Rights, Copyright, Patent, Data use, Phytosanitary, Salvage, Exemption Permit, Material Transfer Agreement, Internationally Recognized Certificate of Compliance, Contract, Memorandum of Understanding, Memorandum of Cooperation, Veterinary Certificate, Human Pathogens, Genetically Modified Organism, Other |- |SpecimenUnit/Loan/Permit |*permitText |The text of a permit related to the gathering/shipping or further details | |- |GGBN/Amplification |amplificationDate |date of amplification; if unknown or general without content you don't have to map it |should be ISO format yyyy-mm-dd |- |GGBN/Amplification |marker | |COX1 |- |GGBN/Amplification |geneticAccessionNumber |the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD; this is a repeatable element, you can provide as much as you want |e.g. AJ45567 |- |GGBN/Amplification |genBankNumber-URI |complete link to the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD | |- |SpecimenUnit |blockedUntil |in case the sample is blocked until a specific date it's nevertheless searchable but customers cannot order it |ISO format |- |SpecimenUnit |blocked |sample is blocked (e.g. because it is consumed), but data are still available |Yes/No |- |GGBN |concentration |concentration of the DNA |65,34 |- |GGBN |@unit |unit of the concentration |µg/ml |- |GGBN |ratioOfAbsorbance260_230 |map only if filled with content |1,2 |- |GGBN |ratioOfAbsorbance260_280 |map only if filled with content |1,8 |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===The voucher identifiers=== {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Associations/UnitAssociation |*UnitID |the UnitID or CatalogueNumber used for GBIF |e.g. the barcode number of your specimens |- |Associations/UnitAssociation |*SourceInstitutionID |the SourceInstitutionID or InstitutionCode used for GBIF |e.g. the acronym of your institution |- |Associations/UnitAssociation |*SourceName |the SourceID or CollectionCode used for GBIF |e.g. the name of the collection where the specimen belongs to, e.g. "Birds" |- |Associations/UnitAssociation |*AssociationType |the Relation between the DNA and the voucher |e.g. "DNA and voucher from same individual" |- |Associations/UnitAssociation |*DatasetAccessPoint |the wrapper url of the voucher record |e.g. "http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar" |} ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} ===Multimedia items of the voucher=== These should be mapped in the specimen mapping, not the DNA mapping. {| class="wikitable" ! Category ! Element ! Discription ! Comment |- | rowspan="3" | Metadata | Title | | |- | IPR | | |- | TechnicalContact | | |- | Unit | UnitID | | |} 200daf6f1854b18ef7be557ea29a1d94368e44b0 1008 1007 2018-09-27T13:58:15Z JoergHoletschek 3 /* General unit data */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" ! width="120px" | Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) | |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. | |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber | | |- |RecordBasis | | |- |KindOfUnit | | |- |Sex | | |- |RecordURI | | |- |Notes | | |- |DateLastEdited | | |- |LastEditor | | |- |} ===The sample identifiers=== We use identifier the same way as GBIF does. In addition to the traditional triple ID (see below) you can also provide the GUID. '''Please note that UnitID should be unique in one database!''' Mostly the extraction number is used for it. {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Unit |*<span style="color:red">SourceID</span> |short description of relevant collection |should be "DNA Bank" or "tissue collection" |- |Unit |*<span style="color:red">SourceInstitutionID</span> |short name/abbreviation of relevant institution |BGBM |- |Unit |*<span style="color:red">UnitID</span> |DNA extraction number or tissue number |DNA 123 |- |Unit |UnitGUID |sample record GUID |f6c7fd5a-2f04-4dcb-ba38-0230bf196d30 |} ===GGBN terms=== The following fields are highly recommended, some are mandatory (marked with *), but feel free to map more! <div id="wikinote">'''Please have a look at the [http://terms.tdwg.org/wiki/GGBN_Data_Standard GGBN Data Standard] documentation where all GGBN terms are described in detail. The table below is just listing the most important ones. Since the Nagoya Protocol has come into force we higly recommend to map the Permit fields. They will become mandatory for providing data to GGBN at the end of 2020. Until then all core members must have implemented the required fields in their databases.'''</div> {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- | |*RecordBasis |The specific nature of the data record, controlled vocabulary! |"PreservedSpecimen", "MaterialSample", "FossilSpecimen", "LivingSpecimen", "HumanObservation", "MachineObservation"; for Tissue or DNA samples use "MaterialSample" |- | |*materialSampleType |Classification of kind of physical sample in addition to BasisOfRecord/RecordBasis and Preparation Type |"tissue", "DNA", "specimen" |- |SpecimenUnit/Preparation |*preparationType |description of type of material, free text |or DNA: gDNA, eDNA, aDNA; for tissues/specimens: leaf, muscle, leg, blood |- |SpecimenUnit/Preparation |*preparationDate |date of DNA extraction |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparatioMaterials |extraction kit or protocoll |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparedBy |extraction staff |if unknown type "unknown" |- |SpecimenUnit/Preservation |preservation |preservation of the tissue or DNA |if unknown type "unknown" |- |SpecimenUnit/Loan/Permit |*permitStatus |Information about the presence, absence or other basic status of permits associated with the sample(s), controlled vocabulary! |Permit available, Permit not required, Permit not available, '''Unknown Material collected after 2014-10-12 cannot be in "Unknown" permit status!''' |- |SpecimenUnit/Loan/Permit |*permitStatusQualifier |Description of why a certain permit was not required or why Permit Status is unknown |"no national requirement for a permit at date of access", "officially authorized illegal holder", "collected on private land", "pre-Nagoya" |- |SpecimenUnit/Loan/Permit |*permitType |A permit is a document that allows someone to take an action that otherwise would not be allowed, controlled vocabulary! |Collecting Permit, Import Permit, Export Permit, Intellectual Property Rights, Copyright, Patent, Data use, Phytosanitary, Salvage, Exemption Permit, Material Transfer Agreement, Internationally Recognized Certificate of Compliance, Contract, Memorandum of Understanding, Memorandum of Cooperation, Veterinary Certificate, Human Pathogens, Genetically Modified Organism, Other |- |SpecimenUnit/Loan/Permit |*permitText |The text of a permit related to the gathering/shipping or further details | |- |GGBN/Amplification |amplificationDate |date of amplification; if unknown or general without content you don't have to map it |should be ISO format yyyy-mm-dd |- |GGBN/Amplification |marker | |COX1 |- |GGBN/Amplification |geneticAccessionNumber |the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD; this is a repeatable element, you can provide as much as you want |e.g. AJ45567 |- |GGBN/Amplification |genBankNumber-URI |complete link to the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD | |- |SpecimenUnit |blockedUntil |in case the sample is blocked until a specific date it's nevertheless searchable but customers cannot order it |ISO format |- |SpecimenUnit |blocked |sample is blocked (e.g. because it is consumed), but data are still available |Yes/No |- |GGBN |concentration |concentration of the DNA |65,34 |- |GGBN |@unit |unit of the concentration |µg/ml |- |GGBN |ratioOfAbsorbance260_230 |map only if filled with content |1,2 |- |GGBN |ratioOfAbsorbance260_280 |map only if filled with content |1,8 |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===The voucher identifiers=== {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Associations/UnitAssociation |*UnitID |the UnitID or CatalogueNumber used for GBIF |e.g. the barcode number of your specimens |- |Associations/UnitAssociation |*SourceInstitutionID |the SourceInstitutionID or InstitutionCode used for GBIF |e.g. the acronym of your institution |- |Associations/UnitAssociation |*SourceName |the SourceID or CollectionCode used for GBIF |e.g. the name of the collection where the specimen belongs to, e.g. "Birds" |- |Associations/UnitAssociation |*AssociationType |the Relation between the DNA and the voucher |e.g. "DNA and voucher from same individual" |- |Associations/UnitAssociation |*DatasetAccessPoint |the wrapper url of the voucher record |e.g. "http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar" |} ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} ===Multimedia items of the voucher=== These should be mapped in the specimen mapping, not the DNA mapping. {| class="wikitable" ! Category ! Element ! Discription ! Comment |- | rowspan="3" | Metadata | Title | | |- | IPR | | |- | TechnicalContact | | |- | Unit | UnitID | | |} 1118b5df4a17925fd61378b9c45bb46dbd6c3966 1009 1008 2018-09-27T14:04:51Z JoergHoletschek 3 /* General unit data */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" ! width="120px" | Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) | |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. | |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |1055 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit | | |- |Sex | | |- |RecordURI | | |- |Notes | | |- |DateLastEdited | | |- |LastEditor | | |- |} ===The sample identifiers=== We use identifier the same way as GBIF does. In addition to the traditional triple ID (see below) you can also provide the GUID. '''Please note that UnitID should be unique in one database!''' Mostly the extraction number is used for it. {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Unit |*<span style="color:red">SourceID</span> |short description of relevant collection |should be "DNA Bank" or "tissue collection" |- |Unit |*<span style="color:red">SourceInstitutionID</span> |short name/abbreviation of relevant institution |BGBM |- |Unit |*<span style="color:red">UnitID</span> |DNA extraction number or tissue number |DNA 123 |- |Unit |UnitGUID |sample record GUID |f6c7fd5a-2f04-4dcb-ba38-0230bf196d30 |} ===GGBN terms=== The following fields are highly recommended, some are mandatory (marked with *), but feel free to map more! <div id="wikinote">'''Please have a look at the [http://terms.tdwg.org/wiki/GGBN_Data_Standard GGBN Data Standard] documentation where all GGBN terms are described in detail. The table below is just listing the most important ones. Since the Nagoya Protocol has come into force we higly recommend to map the Permit fields. They will become mandatory for providing data to GGBN at the end of 2020. Until then all core members must have implemented the required fields in their databases.'''</div> {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- | |*RecordBasis |The specific nature of the data record, controlled vocabulary! |"PreservedSpecimen", "MaterialSample", "FossilSpecimen", "LivingSpecimen", "HumanObservation", "MachineObservation"; for Tissue or DNA samples use "MaterialSample" |- | |*materialSampleType |Classification of kind of physical sample in addition to BasisOfRecord/RecordBasis and Preparation Type |"tissue", "DNA", "specimen" |- |SpecimenUnit/Preparation |*preparationType |description of type of material, free text |or DNA: gDNA, eDNA, aDNA; for tissues/specimens: leaf, muscle, leg, blood |- |SpecimenUnit/Preparation |*preparationDate |date of DNA extraction |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparatioMaterials |extraction kit or protocoll |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparedBy |extraction staff |if unknown type "unknown" |- |SpecimenUnit/Preservation |preservation |preservation of the tissue or DNA |if unknown type "unknown" |- |SpecimenUnit/Loan/Permit |*permitStatus |Information about the presence, absence or other basic status of permits associated with the sample(s), controlled vocabulary! |Permit available, Permit not required, Permit not available, '''Unknown Material collected after 2014-10-12 cannot be in "Unknown" permit status!''' |- |SpecimenUnit/Loan/Permit |*permitStatusQualifier |Description of why a certain permit was not required or why Permit Status is unknown |"no national requirement for a permit at date of access", "officially authorized illegal holder", "collected on private land", "pre-Nagoya" |- |SpecimenUnit/Loan/Permit |*permitType |A permit is a document that allows someone to take an action that otherwise would not be allowed, controlled vocabulary! |Collecting Permit, Import Permit, Export Permit, Intellectual Property Rights, Copyright, Patent, Data use, Phytosanitary, Salvage, Exemption Permit, Material Transfer Agreement, Internationally Recognized Certificate of Compliance, Contract, Memorandum of Understanding, Memorandum of Cooperation, Veterinary Certificate, Human Pathogens, Genetically Modified Organism, Other |- |SpecimenUnit/Loan/Permit |*permitText |The text of a permit related to the gathering/shipping or further details | |- |GGBN/Amplification |amplificationDate |date of amplification; if unknown or general without content you don't have to map it |should be ISO format yyyy-mm-dd |- |GGBN/Amplification |marker | |COX1 |- |GGBN/Amplification |geneticAccessionNumber |the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD; this is a repeatable element, you can provide as much as you want |e.g. AJ45567 |- |GGBN/Amplification |genBankNumber-URI |complete link to the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD | |- |SpecimenUnit |blockedUntil |in case the sample is blocked until a specific date it's nevertheless searchable but customers cannot order it |ISO format |- |SpecimenUnit |blocked |sample is blocked (e.g. because it is consumed), but data are still available |Yes/No |- |GGBN |concentration |concentration of the DNA |65,34 |- |GGBN |@unit |unit of the concentration |µg/ml |- |GGBN |ratioOfAbsorbance260_230 |map only if filled with content |1,2 |- |GGBN |ratioOfAbsorbance260_280 |map only if filled with content |1,8 |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===The voucher identifiers=== {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Associations/UnitAssociation |*UnitID |the UnitID or CatalogueNumber used for GBIF |e.g. the barcode number of your specimens |- |Associations/UnitAssociation |*SourceInstitutionID |the SourceInstitutionID or InstitutionCode used for GBIF |e.g. the acronym of your institution |- |Associations/UnitAssociation |*SourceName |the SourceID or CollectionCode used for GBIF |e.g. the name of the collection where the specimen belongs to, e.g. "Birds" |- |Associations/UnitAssociation |*AssociationType |the Relation between the DNA and the voucher |e.g. "DNA and voucher from same individual" |- |Associations/UnitAssociation |*DatasetAccessPoint |the wrapper url of the voucher record |e.g. "http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar" |} ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} ===Multimedia items of the voucher=== These should be mapped in the specimen mapping, not the DNA mapping. {| class="wikitable" ! Category ! Element ! Discription ! Comment |- | rowspan="3" | Metadata | Title | | |- | IPR | | |- | TechnicalContact | | |- | Unit | UnitID | | |} 58ce317cae0dc139e64f90f1af845b1ba3cf19e9 1010 1009 2018-09-27T14:13:16Z JoergHoletschek 3 /* General unit data */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" ! width="120px" | Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) | |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. | |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |1055 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===The sample identifiers=== We use identifier the same way as GBIF does. In addition to the traditional triple ID (see below) you can also provide the GUID. '''Please note that UnitID should be unique in one database!''' Mostly the extraction number is used for it. {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Unit |*<span style="color:red">SourceID</span> |short description of relevant collection |should be "DNA Bank" or "tissue collection" |- |Unit |*<span style="color:red">SourceInstitutionID</span> |short name/abbreviation of relevant institution |BGBM |- |Unit |*<span style="color:red">UnitID</span> |DNA extraction number or tissue number |DNA 123 |- |Unit |UnitGUID |sample record GUID |f6c7fd5a-2f04-4dcb-ba38-0230bf196d30 |} ===GGBN terms=== The following fields are highly recommended, some are mandatory (marked with *), but feel free to map more! <div id="wikinote">'''Please have a look at the [http://terms.tdwg.org/wiki/GGBN_Data_Standard GGBN Data Standard] documentation where all GGBN terms are described in detail. The table below is just listing the most important ones. Since the Nagoya Protocol has come into force we higly recommend to map the Permit fields. They will become mandatory for providing data to GGBN at the end of 2020. Until then all core members must have implemented the required fields in their databases.'''</div> {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- | |*RecordBasis |The specific nature of the data record, controlled vocabulary! |"PreservedSpecimen", "MaterialSample", "FossilSpecimen", "LivingSpecimen", "HumanObservation", "MachineObservation"; for Tissue or DNA samples use "MaterialSample" |- | |*materialSampleType |Classification of kind of physical sample in addition to BasisOfRecord/RecordBasis and Preparation Type |"tissue", "DNA", "specimen" |- |SpecimenUnit/Preparation |*preparationType |description of type of material, free text |or DNA: gDNA, eDNA, aDNA; for tissues/specimens: leaf, muscle, leg, blood |- |SpecimenUnit/Preparation |*preparationDate |date of DNA extraction |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparatioMaterials |extraction kit or protocoll |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparedBy |extraction staff |if unknown type "unknown" |- |SpecimenUnit/Preservation |preservation |preservation of the tissue or DNA |if unknown type "unknown" |- |SpecimenUnit/Loan/Permit |*permitStatus |Information about the presence, absence or other basic status of permits associated with the sample(s), controlled vocabulary! |Permit available, Permit not required, Permit not available, '''Unknown Material collected after 2014-10-12 cannot be in "Unknown" permit status!''' |- |SpecimenUnit/Loan/Permit |*permitStatusQualifier |Description of why a certain permit was not required or why Permit Status is unknown |"no national requirement for a permit at date of access", "officially authorized illegal holder", "collected on private land", "pre-Nagoya" |- |SpecimenUnit/Loan/Permit |*permitType |A permit is a document that allows someone to take an action that otherwise would not be allowed, controlled vocabulary! |Collecting Permit, Import Permit, Export Permit, Intellectual Property Rights, Copyright, Patent, Data use, Phytosanitary, Salvage, Exemption Permit, Material Transfer Agreement, Internationally Recognized Certificate of Compliance, Contract, Memorandum of Understanding, Memorandum of Cooperation, Veterinary Certificate, Human Pathogens, Genetically Modified Organism, Other |- |SpecimenUnit/Loan/Permit |*permitText |The text of a permit related to the gathering/shipping or further details | |- |GGBN/Amplification |amplificationDate |date of amplification; if unknown or general without content you don't have to map it |should be ISO format yyyy-mm-dd |- |GGBN/Amplification |marker | |COX1 |- |GGBN/Amplification |geneticAccessionNumber |the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD; this is a repeatable element, you can provide as much as you want |e.g. AJ45567 |- |GGBN/Amplification |genBankNumber-URI |complete link to the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD | |- |SpecimenUnit |blockedUntil |in case the sample is blocked until a specific date it's nevertheless searchable but customers cannot order it |ISO format |- |SpecimenUnit |blocked |sample is blocked (e.g. because it is consumed), but data are still available |Yes/No |- |GGBN |concentration |concentration of the DNA |65,34 |- |GGBN |@unit |unit of the concentration |µg/ml |- |GGBN |ratioOfAbsorbance260_230 |map only if filled with content |1,2 |- |GGBN |ratioOfAbsorbance260_280 |map only if filled with content |1,8 |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===The voucher identifiers=== {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Associations/UnitAssociation |*UnitID |the UnitID or CatalogueNumber used for GBIF |e.g. the barcode number of your specimens |- |Associations/UnitAssociation |*SourceInstitutionID |the SourceInstitutionID or InstitutionCode used for GBIF |e.g. the acronym of your institution |- |Associations/UnitAssociation |*SourceName |the SourceID or CollectionCode used for GBIF |e.g. the name of the collection where the specimen belongs to, e.g. "Birds" |- |Associations/UnitAssociation |*AssociationType |the Relation between the DNA and the voucher |e.g. "DNA and voucher from same individual" |- |Associations/UnitAssociation |*DatasetAccessPoint |the wrapper url of the voucher record |e.g. "http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar" |} ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} ===Multimedia items of the voucher=== These should be mapped in the specimen mapping, not the DNA mapping. {| class="wikitable" ! Category ! Element ! Discription ! Comment |- | rowspan="3" | Metadata | Title | | |- | IPR | | |- | TechnicalContact | | |- | Unit | UnitID | | |} c871bafd44267ebbf6e7cfc379755de6a187f034 1011 1010 2018-09-27T14:17:02Z JoergHoletschek 3 /* IPR Statements */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" ! width="120px" | Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |1055 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===The sample identifiers=== We use identifier the same way as GBIF does. In addition to the traditional triple ID (see below) you can also provide the GUID. '''Please note that UnitID should be unique in one database!''' Mostly the extraction number is used for it. {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Unit |*<span style="color:red">SourceID</span> |short description of relevant collection |should be "DNA Bank" or "tissue collection" |- |Unit |*<span style="color:red">SourceInstitutionID</span> |short name/abbreviation of relevant institution |BGBM |- |Unit |*<span style="color:red">UnitID</span> |DNA extraction number or tissue number |DNA 123 |- |Unit |UnitGUID |sample record GUID |f6c7fd5a-2f04-4dcb-ba38-0230bf196d30 |} ===GGBN terms=== The following fields are highly recommended, some are mandatory (marked with *), but feel free to map more! <div id="wikinote">'''Please have a look at the [http://terms.tdwg.org/wiki/GGBN_Data_Standard GGBN Data Standard] documentation where all GGBN terms are described in detail. The table below is just listing the most important ones. Since the Nagoya Protocol has come into force we higly recommend to map the Permit fields. They will become mandatory for providing data to GGBN at the end of 2020. Until then all core members must have implemented the required fields in their databases.'''</div> {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- | |*RecordBasis |The specific nature of the data record, controlled vocabulary! |"PreservedSpecimen", "MaterialSample", "FossilSpecimen", "LivingSpecimen", "HumanObservation", "MachineObservation"; for Tissue or DNA samples use "MaterialSample" |- | |*materialSampleType |Classification of kind of physical sample in addition to BasisOfRecord/RecordBasis and Preparation Type |"tissue", "DNA", "specimen" |- |SpecimenUnit/Preparation |*preparationType |description of type of material, free text |or DNA: gDNA, eDNA, aDNA; for tissues/specimens: leaf, muscle, leg, blood |- |SpecimenUnit/Preparation |*preparationDate |date of DNA extraction |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparatioMaterials |extraction kit or protocoll |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparedBy |extraction staff |if unknown type "unknown" |- |SpecimenUnit/Preservation |preservation |preservation of the tissue or DNA |if unknown type "unknown" |- |SpecimenUnit/Loan/Permit |*permitStatus |Information about the presence, absence or other basic status of permits associated with the sample(s), controlled vocabulary! |Permit available, Permit not required, Permit not available, '''Unknown Material collected after 2014-10-12 cannot be in "Unknown" permit status!''' |- |SpecimenUnit/Loan/Permit |*permitStatusQualifier |Description of why a certain permit was not required or why Permit Status is unknown |"no national requirement for a permit at date of access", "officially authorized illegal holder", "collected on private land", "pre-Nagoya" |- |SpecimenUnit/Loan/Permit |*permitType |A permit is a document that allows someone to take an action that otherwise would not be allowed, controlled vocabulary! |Collecting Permit, Import Permit, Export Permit, Intellectual Property Rights, Copyright, Patent, Data use, Phytosanitary, Salvage, Exemption Permit, Material Transfer Agreement, Internationally Recognized Certificate of Compliance, Contract, Memorandum of Understanding, Memorandum of Cooperation, Veterinary Certificate, Human Pathogens, Genetically Modified Organism, Other |- |SpecimenUnit/Loan/Permit |*permitText |The text of a permit related to the gathering/shipping or further details | |- |GGBN/Amplification |amplificationDate |date of amplification; if unknown or general without content you don't have to map it |should be ISO format yyyy-mm-dd |- |GGBN/Amplification |marker | |COX1 |- |GGBN/Amplification |geneticAccessionNumber |the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD; this is a repeatable element, you can provide as much as you want |e.g. AJ45567 |- |GGBN/Amplification |genBankNumber-URI |complete link to the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD | |- |SpecimenUnit |blockedUntil |in case the sample is blocked until a specific date it's nevertheless searchable but customers cannot order it |ISO format |- |SpecimenUnit |blocked |sample is blocked (e.g. because it is consumed), but data are still available |Yes/No |- |GGBN |concentration |concentration of the DNA |65,34 |- |GGBN |@unit |unit of the concentration |µg/ml |- |GGBN |ratioOfAbsorbance260_230 |map only if filled with content |1,2 |- |GGBN |ratioOfAbsorbance260_280 |map only if filled with content |1,8 |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===The voucher identifiers=== {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Associations/UnitAssociation |*UnitID |the UnitID or CatalogueNumber used for GBIF |e.g. the barcode number of your specimens |- |Associations/UnitAssociation |*SourceInstitutionID |the SourceInstitutionID or InstitutionCode used for GBIF |e.g. the acronym of your institution |- |Associations/UnitAssociation |*SourceName |the SourceID or CollectionCode used for GBIF |e.g. the name of the collection where the specimen belongs to, e.g. "Birds" |- |Associations/UnitAssociation |*AssociationType |the Relation between the DNA and the voucher |e.g. "DNA and voucher from same individual" |- |Associations/UnitAssociation |*DatasetAccessPoint |the wrapper url of the voucher record |e.g. "http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar" |} ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} ===Multimedia items of the voucher=== These should be mapped in the specimen mapping, not the DNA mapping. {| class="wikitable" ! Category ! Element ! Discription ! Comment |- | rowspan="3" | Metadata | Title | | |- | IPR | | |- | TechnicalContact | | |- | Unit | UnitID | | |} ce8577e1bed8eadc8234d5810369f7d75c313686 1012 1011 2018-09-27T14:18:45Z JoergHoletschek 3 /* The sample identifiers */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" ! width="120px" | Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |1055 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===GGBN terms=== The following fields are highly recommended, some are mandatory (marked with *), but feel free to map more! <div id="wikinote">'''Please have a look at the [http://terms.tdwg.org/wiki/GGBN_Data_Standard GGBN Data Standard] documentation where all GGBN terms are described in detail. The table below is just listing the most important ones. Since the Nagoya Protocol has come into force we higly recommend to map the Permit fields. They will become mandatory for providing data to GGBN at the end of 2020. Until then all core members must have implemented the required fields in their databases.'''</div> {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- | |*RecordBasis |The specific nature of the data record, controlled vocabulary! |"PreservedSpecimen", "MaterialSample", "FossilSpecimen", "LivingSpecimen", "HumanObservation", "MachineObservation"; for Tissue or DNA samples use "MaterialSample" |- | |*materialSampleType |Classification of kind of physical sample in addition to BasisOfRecord/RecordBasis and Preparation Type |"tissue", "DNA", "specimen" |- |SpecimenUnit/Preparation |*preparationType |description of type of material, free text |or DNA: gDNA, eDNA, aDNA; for tissues/specimens: leaf, muscle, leg, blood |- |SpecimenUnit/Preparation |*preparationDate |date of DNA extraction |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparatioMaterials |extraction kit or protocoll |if unknown type "unknown" |- |SpecimenUnit/Preparation |*preparedBy |extraction staff |if unknown type "unknown" |- |SpecimenUnit/Preservation |preservation |preservation of the tissue or DNA |if unknown type "unknown" |- |SpecimenUnit/Loan/Permit |*permitStatus |Information about the presence, absence or other basic status of permits associated with the sample(s), controlled vocabulary! |Permit available, Permit not required, Permit not available, '''Unknown Material collected after 2014-10-12 cannot be in "Unknown" permit status!''' |- |SpecimenUnit/Loan/Permit |*permitStatusQualifier |Description of why a certain permit was not required or why Permit Status is unknown |"no national requirement for a permit at date of access", "officially authorized illegal holder", "collected on private land", "pre-Nagoya" |- |SpecimenUnit/Loan/Permit |*permitType |A permit is a document that allows someone to take an action that otherwise would not be allowed, controlled vocabulary! |Collecting Permit, Import Permit, Export Permit, Intellectual Property Rights, Copyright, Patent, Data use, Phytosanitary, Salvage, Exemption Permit, Material Transfer Agreement, Internationally Recognized Certificate of Compliance, Contract, Memorandum of Understanding, Memorandum of Cooperation, Veterinary Certificate, Human Pathogens, Genetically Modified Organism, Other |- |SpecimenUnit/Loan/Permit |*permitText |The text of a permit related to the gathering/shipping or further details | |- |GGBN/Amplification |amplificationDate |date of amplification; if unknown or general without content you don't have to map it |should be ISO format yyyy-mm-dd |- |GGBN/Amplification |marker | |COX1 |- |GGBN/Amplification |geneticAccessionNumber |the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD; this is a repeatable element, you can provide as much as you want |e.g. AJ45567 |- |GGBN/Amplification |genBankNumber-URI |complete link to the accession number of NCBI/EMBL/DDBJ or the process ID of BOLD | |- |SpecimenUnit |blockedUntil |in case the sample is blocked until a specific date it's nevertheless searchable but customers cannot order it |ISO format |- |SpecimenUnit |blocked |sample is blocked (e.g. because it is consumed), but data are still available |Yes/No |- |GGBN |concentration |concentration of the DNA |65,34 |- |GGBN |@unit |unit of the concentration |µg/ml |- |GGBN |ratioOfAbsorbance260_230 |map only if filled with content |1,2 |- |GGBN |ratioOfAbsorbance260_280 |map only if filled with content |1,8 |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===The voucher identifiers=== {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Associations/UnitAssociation |*UnitID |the UnitID or CatalogueNumber used for GBIF |e.g. the barcode number of your specimens |- |Associations/UnitAssociation |*SourceInstitutionID |the SourceInstitutionID or InstitutionCode used for GBIF |e.g. the acronym of your institution |- |Associations/UnitAssociation |*SourceName |the SourceID or CollectionCode used for GBIF |e.g. the name of the collection where the specimen belongs to, e.g. "Birds" |- |Associations/UnitAssociation |*AssociationType |the Relation between the DNA and the voucher |e.g. "DNA and voucher from same individual" |- |Associations/UnitAssociation |*DatasetAccessPoint |the wrapper url of the voucher record |e.g. "http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar" |} ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} ===Multimedia items of the voucher=== These should be mapped in the specimen mapping, not the DNA mapping. {| class="wikitable" ! Category ! Element ! Discription ! Comment |- | rowspan="3" | Metadata | Title | | |- | IPR | | |- | TechnicalContact | | |- | Unit | UnitID | | |} 1290099ed1500c433865e0eaf6fa614e1c3f32ea 1013 1012 2018-09-27T14:18:53Z JoergHoletschek 3 /* GGBN terms */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" ! width="120px" | Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |1055 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===The voucher identifiers=== {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Associations/UnitAssociation |*UnitID |the UnitID or CatalogueNumber used for GBIF |e.g. the barcode number of your specimens |- |Associations/UnitAssociation |*SourceInstitutionID |the SourceInstitutionID or InstitutionCode used for GBIF |e.g. the acronym of your institution |- |Associations/UnitAssociation |*SourceName |the SourceID or CollectionCode used for GBIF |e.g. the name of the collection where the specimen belongs to, e.g. "Birds" |- |Associations/UnitAssociation |*AssociationType |the Relation between the DNA and the voucher |e.g. "DNA and voucher from same individual" |- |Associations/UnitAssociation |*DatasetAccessPoint |the wrapper url of the voucher record |e.g. "http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar" |} ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} ===Multimedia items of the voucher=== These should be mapped in the specimen mapping, not the DNA mapping. {| class="wikitable" ! Category ! Element ! Discription ! Comment |- | rowspan="3" | Metadata | Title | | |- | IPR | | |- | TechnicalContact | | |- | Unit | UnitID | | |} fd0467ae89b589781bdf3ada504ffab4e6844809 1014 1013 2018-09-27T14:19:25Z JoergHoletschek 3 /* Multimedia items of the voucher */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" ! width="120px" | Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |1055 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===The voucher identifiers=== {| class="wikitable" ! width="120px" | Group !Element !Remarks !Example |- |Associations/UnitAssociation |*UnitID |the UnitID or CatalogueNumber used for GBIF |e.g. the barcode number of your specimens |- |Associations/UnitAssociation |*SourceInstitutionID |the SourceInstitutionID or InstitutionCode used for GBIF |e.g. the acronym of your institution |- |Associations/UnitAssociation |*SourceName |the SourceID or CollectionCode used for GBIF |e.g. the name of the collection where the specimen belongs to, e.g. "Birds" |- |Associations/UnitAssociation |*AssociationType |the Relation between the DNA and the voucher |e.g. "DNA and voucher from same individual" |- |Associations/UnitAssociation |*DatasetAccessPoint |the wrapper url of the voucher record |e.g. "http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar" |} ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} 372ee588656823961ec5e5fbda013456de6bddcc 1015 1014 2018-09-27T14:19:40Z JoergHoletschek 3 /* The voucher identifiers */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" ! width="120px" | Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |1055 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} c78677dab7cfd5f83c430e10e53018fcdef464a6 1016 1015 2018-09-27T14:21:29Z JoergHoletschek 3 /* General metadata */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" ! width="120px" | Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |1055 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} 061bf0c321e429b9a63c68d26615d98875c58afc 1017 1016 2018-09-27T14:28:30Z JoergHoletschek 3 /* Specimen Data */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" ! width="120px" | Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |1055 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} ===XXX=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} de67bf9affb902867916e7affb1e0938aa94e7b6 1018 1017 2018-09-27T14:41:16Z JoergHoletschek 3 /* Botanical Garden Unit */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" ! width="120px" | Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |1055 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===XXX=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} 7fb3989051a989f5360e97b730bf3ed38ebed7a4 1019 1018 2018-09-27T14:42:12Z JoergHoletschek 3 /* General metadata */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |1055 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===XXX=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} ==Related Specimen Data== <div id="wikinote">All specimen voucher information must be available via a GBIF compliant database! Both Darwin Core-Archive and BioCASE specimen providers are possibe! To provide the underlying specimen data you can use your existing GBIF dataset mapping.</div> [[File:Relationships.jpg|thumb|500px|In the database and the mapping the child record should refer to its mother record in opposite to the real world example]] ===Gathering event of the voucher=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} 1e5b0a1ebaf779de9562c6d5cde01742191fa7d3 1020 1019 2018-09-28T07:01:34Z JoergHoletschek 3 /* Related Specimen Data */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |1055 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===XXX=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} 0e5fed79c9b811cfb014b7e0268b4ce4ca99e590 0 115 1021 1020 2018-09-28T07:02:07Z JoergHoletschek 3 /* General unit data */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |1055 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== '''All elements marked with * will be indexed and must be mapped!''' Mapping these gathering facts twice (one for the specimen database and one for the DNA mapping) is required because of indexing and later search purposes. {| class="wikitable" ! | Group !Element !Remarks !Example |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===XXX=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} e0ad46a8bf264a8db521b2a3259dbda5a9bce9a4 1022 1021 2018-09-28T07:04:20Z JoergHoletschek 3 /* Gathering event */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |1055 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' sub section of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remarks |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===XXX=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} 34f662d7508b8e17c93f07a93df7146809a0bd36 1023 1022 2018-09-28T07:04:37Z JoergHoletschek 3 /* General unit data */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' sub section of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remarks |- | |*CollectorsFieldNumber |the number the collector gave to the specimen in the field, often used in Botany but not in Zoology; map it if you have content |e.g. 765/10 |- |Gathering/Agents/GatheringAgent |*FullName |the Collector or Collector Team |e.g. Scholz & Sipman |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===XXX=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} c1afab57a9933bdcb5c75767a58ae76821a90b83 1024 1023 2018-09-28T07:11:36Z JoergHoletschek 3 /* Gathering event */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remarks |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' sub section of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remarks |- |Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |Agents |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===XXX=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} 7ed6df453a8e7be2506ef85f712ba741cecf242a 1025 1024 2018-09-28T07:12:05Z JoergHoletschek 3 /* General metadata */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' sub section of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remarks |- |Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |Agents |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===XXX=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} 9e3900e86f0b82dad468be7fab98ccca58a0cbaa 1026 1025 2018-09-28T07:12:29Z JoergHoletschek 3 /* Gathering event */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' sub section of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |Gathering/Altitude |*LowerValue |if you have both the lower and upper value in different columns map both field, if not map LowerValue only |e.g. 100 |- |Gathering/Altitude |UpperValue | |e.g. 200 |- |Gathering/Altitude |*Unit | |e.g. m |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===XXX=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} 0297da76e1c1b8a35a2fc85b45a11f7c4ac78cfb 1027 1026 2018-09-28T07:21:34Z JoergHoletschek 3 /* Gathering event */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' sub section of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===XXX=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} cf76a4434ef74d48ba99d1c39ebabe4eb4f58b74 1028 1027 2018-09-28T07:23:30Z JoergHoletschek 3 /* Gathering event */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' sub section of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- |rowspan="2" colspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===XXX=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} 8bd29333fdc44173b400d8d997f198530965e89d 1029 1028 2018-09-28T07:27:11Z JoergHoletschek 3 /* Gathering event */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' sub section of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- |rowspan="2" colspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude | |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |rowspan="3" | MeasurementOrFactAtomised |LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===XXX=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} 92463bc558aad8bf2bf268d6617ba88c724e26bd 1030 1029 2018-09-28T07:30:34Z JoergHoletschek 3 /* Gathering event */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' sub section of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |Gathering/Country |*ISO3166Code |ISO code of the country where the voucher was collected |e.g. US |- |Gathering/Country |*Name |english Name of the country |e.g. United States of America |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===XXX=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} 481df86a528998ef78fdfc857f6ec813d4abc231 1031 1030 2018-09-28T07:33:40Z JoergHoletschek 3 /* Gathering event */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' sub section of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |Gathering/DateTime |*DateText |date when voucher was collected, if you have content yuo can also use ISO format |e.g. 21. April 1951 |- |Gathering/Locality |*LocalityText | |e.g. 5km NO Berlin |- |Gathering/NamedArea |AreaName |name of continent |e.g. Europe |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===XXX=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} 66cc20f3d0888ad11156f5fb4d0b7307f6283c6d 1032 1031 2018-09-28T07:39:32Z JoergHoletschek 3 /* Gathering event */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- |Gathering/NamedArea |@language |language of the name of continent |e.g. "en" |- |Gathering/SiteCoordinates |*LatitudeDecimal | |e.g. -15,88876 |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===XXX=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} 93f898f9f21f217ed37546d6e75920f4c8beb5f3 1033 1032 2018-09-28T07:43:51Z JoergHoletschek 3 /* Gathering event */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Gathering/SiteCoordinates |*LongitudeDecimal | |e.g. 72,88876 |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===XXX=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} f7024c8fc4509173da2317f10ec0d47776bcbfdb 1034 1033 2018-09-28T08:02:46Z JoergHoletschek 3 /* Gathering event */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |rowspan="8" | SiteCoordinateSets/SiteCoordinates |CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===XXX=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} a9bb1fb9b354530ecf7d5e2f628cca4f3e14615a 1035 1034 2018-09-28T08:05:47Z JoergHoletschek 3 /* Gathering event */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |rowspan="8" | SiteCoordinateSets/SiteCoordinates /CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===XXX=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} 65d555ac0723cf76c792d1fb0217328ed53aa571 1036 1035 2018-09-28T08:06:34Z JoergHoletschek 3 /* Gathering event */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===XXX=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} 1a85e27e29139c290ee8e5338be8de6227ab83b8 1037 1036 2018-09-28T08:32:05Z JoergHoletschek 3 /* Gathering event */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===XXX=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} 2c7d50fc1ef28be5c09f3291c7ad6d6745c2e2ba 1038 1037 2018-09-28T08:38:55Z JoergHoletschek 3 /* Gathering event */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- |AreaDetail | | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===XXX=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} f880e78429e4d3cad836811d468d12ff934306fc 1039 1038 2018-09-28T08:41:54Z JoergHoletschek 3 /* Gathering event */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- |AreaDetail | | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===XXX=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} 852a2ef5c9cbf6b67c1b4dab41211adde6fdc3bc 1040 1039 2018-09-28T08:44:13Z JoergHoletschek 3 /* Gathering event */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===XXX=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} ===Identification history of the voucher=== Mostly specimen databases record the complete determination or identification history of a single specimen. For GGBN we try to get all available information into the portal. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} 977cff4a36dd5e4f268c54735403dbd605f60eff 1041 1040 2018-09-28T09:00:29Z JoergHoletschek 3 /* Identification history of the voucher */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===XXX=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} e5dfab7fbb3bbbe1228233a12ddf4da07fb83c62 1042 1041 2018-09-28T09:02:32Z JoergHoletschek 3 /* Botanical Garden Unit */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" ! | Group !Element !Remarks !Example |- |Identification |*PreferredFlag |mark the presently preferred Identification |e.g. true, false, 0, 1 |- |Identification/HigherTaxon |*HigherTaxonName |the name of the higher taxon, please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |e.g. Asteraceae, Animalia |- |Identification/HigherTaxon |*HigherTaxonRank |the rank of the taxon in english or latin |e.g. familia, regnum, phylum |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===XXX=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} 4c8af43741cd651442a4668cc84cf70feca1a287 1043 1042 2018-09-28T09:09:46Z JoergHoletschek 3 /* Identification history */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |rowspan="2" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===XXX=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} c499937c3d424b4c8998ddd82ba3fffb16329a89 1044 1043 2018-09-28T09:14:29Z JoergHoletschek 3 /* Identification history */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===XXX=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} 9a7bf6bfef3400e1c80bb9452d5c4a732f3a1ff9 1045 1044 2018-09-28T09:22:03Z JoergHoletschek 3 /* Identification history */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="2" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===XXX=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} 82a6a487bd063b61e2734ba42f429a9723c24904 1046 1045 2018-09-28T09:37:20Z JoergHoletschek 3 /* Identification history */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="2" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |Identification/ScientificName |*FullScientificName |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===XXX=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} 4437915e14689b30290c2dbab221b3041e9a3617 1047 1046 2018-09-28T09:44:30Z JoergHoletschek 3 /* Identification history */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="7" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |rowspan="5" | ScientificName |FullScientificNameString* |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===XXX=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} 5eb8f8d728835b7b664948501035104d169cb1e2 1048 1047 2018-09-28T09:48:37Z JoergHoletschek 3 /* Identification history */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="8" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |rowspan="5" | ScientificName |FullScientificNameString* |the complete name of the taxon including Authors (and years for animals) |e.g. Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |- | - |NameComments | |Free text comments related to the name identified for the unit. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===XXX=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} 48bcb20fa3fc76e5582e63fb94c1050b3d52ce58 1049 1048 2018-09-28T09:50:40Z JoergHoletschek 3 /* Identification history */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="8" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |rowspan="5" | ScientificName |FullScientificNameString* |Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |Complete name of the taxon including Authors (and years for animals). |- |Identification/ScientificName/NameAtomised |*FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*GenusOrMonomial | |e.g. Aaronsohnia |- |Identification/ScientificName/NameAtomised |*InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Identification/ScientificName/NameAtomised |*Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |- | - |NameComments | |Free text comments related to the name identified for the unit. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===XXX=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} 8a7ec9dd338561198c15eec7466b35e59209df27 1050 1049 2018-09-28T09:55:19Z JoergHoletschek 3 /* Identification history */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="8" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |ScientificName |FullScientificNameString* |Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |Complete name of the taxon including Authors (and years for animals). |- |rowspan="4" | ScientificName/NameAtomised/Botanical |GenusOrMonomial | |e.g. Aaronsohnia |- |FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |- | - |NameComments | |Free text comments related to the name identified for the unit. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===XXX=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} c2d5b376c7dbd3df33c08f236170faaed24bf961 1051 1050 2018-09-28T09:56:23Z JoergHoletschek 3 /* Identification history */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="9" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |ScientificName |FullScientificNameString* |Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |Complete name of the taxon including Authors (and years for animals). |- |rowspan="5" | ScientificName/NameAtomised/Botanical |GenusOrMonomial | |e.g. Aaronsohnia |- |FirstEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |InfraspecificEpithet |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. factorovskyi |- |Rank |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |- |AuthorTeam |Please note: ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |e.g. var. |- | - |NameComments | |Free text comments related to the name identified for the unit. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===XXX=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} e4e668a652858d1764ec272082bd5aeaa76a6c30 1052 1051 2018-09-28T10:00:05Z JoergHoletschek 3 /* Identification history */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="9" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |ScientificName |FullScientificNameString* |Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |Complete name of the taxon including Authors (and years for animals). |- |rowspan="5" | ScientificName/NameAtomised/Botanical |GenusOrMonomial |Aaronsohnia |rowspan="5" | ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |- |FirstEpithet |factorovskyi |- |InfraspecificEpithet |factorovskyi |- |Rank |var. |- |AuthorTeam |Warb. & Eig. |- | - |NameComments | |Free text comments related to the name identified for the unit. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===XXX=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |} 4f4d6cc09d12232b600949cbe5cd764af8636e4d 1053 1052 2018-09-28T12:35:39Z JoergHoletschek 3 /* XXX */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="9" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |ScientificName |FullScientificNameString* |Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |Complete name of the taxon including Authors (and years for animals). |- |rowspan="5" | ScientificName/NameAtomised/Botanical |GenusOrMonomial |Aaronsohnia |rowspan="5" | ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |- |FirstEpithet |factorovskyi |- |InfraspecificEpithet |factorovskyi |- |Rank |var. |- |AuthorTeam |Warb. & Eig. |- | - |NameComments | |Free text comments related to the name identified for the unit. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} 57a452f8565d8eefd89bbe241ffde7b119b08274 1054 1053 2018-09-28T12:42:53Z JoergHoletschek 3 /* Botanical Garden Unit */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="9" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |ScientificName |FullScientificNameString* |Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |Complete name of the taxon including Authors (and years for animals). |- |rowspan="5" | ScientificName/NameAtomised/Botanical |GenusOrMonomial |Aaronsohnia |rowspan="5" | ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |- |FirstEpithet |factorovskyi |- |InfraspecificEpithet |factorovskyi |- |Rank |var. |- |AuthorTeam |Warb. & Eig. |- | - |NameComments | |Free text comments related to the name identified for the unit. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===Multimedia=== For providing links to associated images, videos and sounds, the sub-tree ''MultimediaObjects'' can be used. Each ABCD unit can have multiple multimedia objects, each represented by its own ''MultimediaObject'' with the following elements: {| class="wikitable" !ABCD element !Example !Remark |- |FileURI* | | |- |ProductURI* | | |- |Comment | | |- |Format | | |- |FileSize | | |- |Creator | | |- |CreatedDate | | |- |ID | | |- |IPR | | |- |} 2e59e4e42fa937220ae2a8129b6899f2b9053393 1055 1054 2018-09-28T12:55:18Z JoergHoletschek 3 /* Multimedia */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="9" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |ScientificName |FullScientificNameString* |Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |Complete name of the taxon including Authors (and years for animals). |- |rowspan="5" | ScientificName/NameAtomised/Botanical |GenusOrMonomial |Aaronsohnia |rowspan="5" | ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |- |FirstEpithet |factorovskyi |- |InfraspecificEpithet |factorovskyi |- |Rank |var. |- |AuthorTeam |Warb. & Eig. |- | - |NameComments | |Free text comments related to the name identified for the unit. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===Multimedia=== For providing links to associated images, videos and sounds, the sub-tree ''MultimediaObjects'' can be used. Each ABCD unit can have multiple multimedia objects, each represented by its own ''MultimediaObject'' with the following elements: {| class="wikitable" !ABCD element !Example !Remark |- |FileURI* |http://ww2.bgbm.org/herbarium/images/B/-W/14/61/B_-W_14616%20-00%200.jpg |The Universal Resource Identifier (URI) of the underlying multimedia file. |- |ProductURI* |http://ww2.bgbm.org/herbarium/view_biocase.cfm?SpecimenPK=726 |The Universal Resource Identifier of an electronic source of a multimedia object including HTML, Javascript or similar means delivering rendered output. |- |Comment |Image of herbarium specimen E00732880 by Specimen Digitisation Pipeline. |Short description of the multimedia object. |- |Format |image/jpeg |A code or descriptive term for the file or object format. |- |FileSize |2365 |The size in Kb of the file referred to by the URI or filename for the multimedia object. |- |Creator |W.-H. Kusber |The identifier (e.g. name or other id) of personnel who has taken the image. |- |CreatedDate |2012-07-07 |The date the multimedia object was created. |- |ID |22658 |Identifier for the multimedia object relating to or forming the unit. This may be a filename. |- |IPR | |Intellectual Property Rights relating specifically to the multimedia object. This subtree has the same elements as ''Metadata/IPRStatements''. Fore more details, see the remarks in [[asdfds]]. |- |} 96561d932d0debec58795d6e8157f442506e21b7 1056 1055 2018-09-28T12:57:05Z JoergHoletschek 3 /* Multimedia */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="9" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |ScientificName |FullScientificNameString* |Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |Complete name of the taxon including Authors (and years for animals). |- |rowspan="5" | ScientificName/NameAtomised/Botanical |GenusOrMonomial |Aaronsohnia |rowspan="5" | ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |- |FirstEpithet |factorovskyi |- |InfraspecificEpithet |factorovskyi |- |Rank |var. |- |AuthorTeam |Warb. & Eig. |- | - |NameComments | |Free text comments related to the name identified for the unit. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===Multimedia=== For providing links to associated images, videos and sounds, the sub-tree ''MultimediaObjects'' can be used. Each ABCD unit can have multiple multimedia objects, each represented by its own ''MultimediaObject'' with the following elements: {| class="wikitable" !ABCD element !Example !Remark |- |FileURI* |http://ww2.bgbm.org/herbarium/images/B/-W/14/61/B_-W_14616%20-00%200.jpg |The Universal Resource Identifier (URI) of the underlying multimedia file. |- |ProductURI |http://ww2.bgbm.org/herbarium/view_biocase.cfm?SpecimenPK=726 |The Universal Resource Identifier of an electronic source of a multimedia object including HTML, Javascript or similar means delivering rendered output. |- |Comment |Image of herbarium specimen E00732880 by Specimen Digitisation Pipeline. |Short description of the multimedia object. |- |Format |image/jpeg |A code or descriptive term for the file or object format. |- |FileSize |2365 |The size in Kb of the file referred to by the URI or filename for the multimedia object. |- |Creator |W.-H. Kusber |The identifier (e.g. name or other id) of personnel who has taken the image. |- |CreatedDate |2012-07-07 |The date the multimedia object was created. |- |ID |22658 |Identifier for the multimedia object relating to or forming the unit. This may be a filename. |- |IPR | |Intellectual Property Rights relating specifically to the multimedia object. This subtree has the same elements as ''Metadata/IPRStatements''. Fore more details, see the respective [[ABCD2Concepts#General_metadata | remarks]]. |- |} ea6abede60f73941b3c2cec805b3176c30eef5ab 1057 1056 2018-09-28T13:00:45Z JoergHoletschek 3 /* Multimedia */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="9" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |ScientificName |FullScientificNameString* |Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |Complete name of the taxon including Authors (and years for animals). |- |rowspan="5" | ScientificName/NameAtomised/Botanical |GenusOrMonomial |Aaronsohnia |rowspan="5" | ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |- |FirstEpithet |factorovskyi |- |InfraspecificEpithet |factorovskyi |- |Rank |var. |- |AuthorTeam |Warb. & Eig. |- | - |NameComments | |Free text comments related to the name identified for the unit. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===Multimedia=== For providing links to associated images, videos and sounds, the sub-tree ''MultimediaObjects'' can be used. Each ABCD unit can have multiple multimedia objects, each represented by its own ''MultimediaObject'' with the following elements: {| class="wikitable" !ABCD element !Example !Remark |- |FileURI* |http://ww2.bgbm.org/herbarium/images/B/-W/14/61/B_-W_14616%20-00%200.jpg |The Universal Resource Identifier (URI) of the underlying multimedia file. |- |ProductURI |http://ww2.bgbm.org/herbarium/view_biocase.cfm?SpecimenPK=726 |The Universal Resource Identifier of an electronic source of a multimedia object including HTML, Javascript or similar means delivering rendered output. |- |Comment |Image of herbarium specimen E00732880 by Specimen Digitisation Pipeline. |Short description of the multimedia object. |- |Format |image/jpeg |A code or descriptive term for the file or object format. |- |FileSize |2365 |The size in Kb of the file referred to by the URI or filename for the multimedia object. |- |Creator |Paris, Museum National d'Histoire Naturelle, OD |The identifier (e.g. name or other id) of personnel who has taken the image. |- |CreatedDate |2012-07-07 |The date the multimedia object was created. |- |ID |22658 |Identifier for the multimedia object relating to or forming the unit. This may be a filename. |- |IPR | |Intellectual Property Rights relating specifically to the multimedia object. This subtree has the same elements as ''Metadata/IPRStatements''. Fore more details, see the respective [[ABCD2Concepts#General_metadata | remarks]]. |- |} ca297ade236bfa498ced5a4016c745a97320b0a4 1059 1057 2018-10-01T08:27:55Z JoergHoletschek 3 /* IPR Statements */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} '''Note:''' If you publish your data to GBIF, you are required to specify a license in the ''License'' node with the ''language'' attribute and the elements ''Text'' and ''URI''. The latter should contain a link to a well-defined license, for example https://creativecommons.org/publicdomain/zero/1.0/. More details on GBIF licensing can be found [http://www.gbif.org/terms/licences here]. ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="9" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |ScientificName |FullScientificNameString* |Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |Complete name of the taxon including Authors (and years for animals). |- |rowspan="5" | ScientificName/NameAtomised/Botanical |GenusOrMonomial |Aaronsohnia |rowspan="5" | ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |- |FirstEpithet |factorovskyi |- |InfraspecificEpithet |factorovskyi |- |Rank |var. |- |AuthorTeam |Warb. & Eig. |- | - |NameComments | |Free text comments related to the name identified for the unit. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===Multimedia=== For providing links to associated images, videos and sounds, the sub-tree ''MultimediaObjects'' can be used. Each ABCD unit can have multiple multimedia objects, each represented by its own ''MultimediaObject'' with the following elements: {| class="wikitable" !ABCD element !Example !Remark |- |FileURI* |http://ww2.bgbm.org/herbarium/images/B/-W/14/61/B_-W_14616%20-00%200.jpg |The Universal Resource Identifier (URI) of the underlying multimedia file. |- |ProductURI |http://ww2.bgbm.org/herbarium/view_biocase.cfm?SpecimenPK=726 |The Universal Resource Identifier of an electronic source of a multimedia object including HTML, Javascript or similar means delivering rendered output. |- |Comment |Image of herbarium specimen E00732880 by Specimen Digitisation Pipeline. |Short description of the multimedia object. |- |Format |image/jpeg |A code or descriptive term for the file or object format. |- |FileSize |2365 |The size in Kb of the file referred to by the URI or filename for the multimedia object. |- |Creator |Paris, Museum National d'Histoire Naturelle, OD |The identifier (e.g. name or other id) of personnel who has taken the image. |- |CreatedDate |2012-07-07 |The date the multimedia object was created. |- |ID |22658 |Identifier for the multimedia object relating to or forming the unit. This may be a filename. |- |IPR | |Intellectual Property Rights relating specifically to the multimedia object. This subtree has the same elements as ''Metadata/IPRStatements''. Fore more details, see the respective [[ABCD2Concepts#General_metadata | remarks]]. |- |} 95cffc981f6ef83a1b3449d70e4ed5f05b4052b1 1060 1059 2018-10-01T08:30:59Z JoergHoletschek 3 /* General metadata */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact* |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact* |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData* |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation* |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} '''Note:''' If you publish your data to GBIF, you are required to specify a license in the ''License'' node with the ''language'' attribute and the elements ''Text'' and ''URI''. The latter should contain a link to a well-defined license, for example https://creativecommons.org/publicdomain/zero/1.0/. More details on GBIF licensing can be found [http://www.gbif.org/terms/licences here]. ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="9" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |ScientificName |FullScientificNameString* |Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |Complete name of the taxon including Authors (and years for animals). |- |rowspan="5" | ScientificName/NameAtomised/Botanical |GenusOrMonomial |Aaronsohnia |rowspan="5" | ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |- |FirstEpithet |factorovskyi |- |InfraspecificEpithet |factorovskyi |- |Rank |var. |- |AuthorTeam |Warb. & Eig. |- | - |NameComments | |Free text comments related to the name identified for the unit. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===Multimedia=== For providing links to associated images, videos and sounds, the sub-tree ''MultimediaObjects'' can be used. Each ABCD unit can have multiple multimedia objects, each represented by its own ''MultimediaObject'' with the following elements: {| class="wikitable" !ABCD element !Example !Remark |- |FileURI* |http://ww2.bgbm.org/herbarium/images/B/-W/14/61/B_-W_14616%20-00%200.jpg |The Universal Resource Identifier (URI) of the underlying multimedia file. |- |ProductURI |http://ww2.bgbm.org/herbarium/view_biocase.cfm?SpecimenPK=726 |The Universal Resource Identifier of an electronic source of a multimedia object including HTML, Javascript or similar means delivering rendered output. |- |Comment |Image of herbarium specimen E00732880 by Specimen Digitisation Pipeline. |Short description of the multimedia object. |- |Format |image/jpeg |A code or descriptive term for the file or object format. |- |FileSize |2365 |The size in Kb of the file referred to by the URI or filename for the multimedia object. |- |Creator |Paris, Museum National d'Histoire Naturelle, OD |The identifier (e.g. name or other id) of personnel who has taken the image. |- |CreatedDate |2012-07-07 |The date the multimedia object was created. |- |ID |22658 |Identifier for the multimedia object relating to or forming the unit. This may be a filename. |- |IPR | |Intellectual Property Rights relating specifically to the multimedia object. This subtree has the same elements as ''Metadata/IPRStatements''. Fore more details, see the respective [[ABCD2Concepts#General_metadata | remarks]]. |- |} 45b2c95eb30705305652d539c4a73012b5426e8b 1061 1060 2018-10-01T08:31:42Z JoergHoletschek 3 /* Metadata */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens/observations. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact* |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact* |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData* |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation* |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} '''Note:''' If you publish your data to GBIF, you are required to specify a license in the ''License'' node with the ''language'' attribute and the elements ''Text'' and ''URI''. The latter should contain a link to a well-defined license, for example https://creativecommons.org/publicdomain/zero/1.0/. More details on GBIF licensing can be found [http://www.gbif.org/terms/licences here]. ==Specimen Data== In ACBD terminology, a specimen is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="9" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |ScientificName |FullScientificNameString* |Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |Complete name of the taxon including Authors (and years for animals). |- |rowspan="5" | ScientificName/NameAtomised/Botanical |GenusOrMonomial |Aaronsohnia |rowspan="5" | ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |- |FirstEpithet |factorovskyi |- |InfraspecificEpithet |factorovskyi |- |Rank |var. |- |AuthorTeam |Warb. & Eig. |- | - |NameComments | |Free text comments related to the name identified for the unit. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===Multimedia=== For providing links to associated images, videos and sounds, the sub-tree ''MultimediaObjects'' can be used. Each ABCD unit can have multiple multimedia objects, each represented by its own ''MultimediaObject'' with the following elements: {| class="wikitable" !ABCD element !Example !Remark |- |FileURI* |http://ww2.bgbm.org/herbarium/images/B/-W/14/61/B_-W_14616%20-00%200.jpg |The Universal Resource Identifier (URI) of the underlying multimedia file. |- |ProductURI |http://ww2.bgbm.org/herbarium/view_biocase.cfm?SpecimenPK=726 |The Universal Resource Identifier of an electronic source of a multimedia object including HTML, Javascript or similar means delivering rendered output. |- |Comment |Image of herbarium specimen E00732880 by Specimen Digitisation Pipeline. |Short description of the multimedia object. |- |Format |image/jpeg |A code or descriptive term for the file or object format. |- |FileSize |2365 |The size in Kb of the file referred to by the URI or filename for the multimedia object. |- |Creator |Paris, Museum National d'Histoire Naturelle, OD |The identifier (e.g. name or other id) of personnel who has taken the image. |- |CreatedDate |2012-07-07 |The date the multimedia object was created. |- |ID |22658 |Identifier for the multimedia object relating to or forming the unit. This may be a filename. |- |IPR | |Intellectual Property Rights relating specifically to the multimedia object. This subtree has the same elements as ''Metadata/IPRStatements''. Fore more details, see the respective [[ABCD2Concepts#General_metadata | remarks]]. |- |} d06c9cdf1fd5444830b7b12f2205ff99bfed86da 1062 1061 2018-10-01T08:37:30Z JoergHoletschek 3 /* Specimen Data */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens/observations. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact* |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact* |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData* |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation* |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} '''Note:''' If you publish your data to GBIF, you are required to specify a license in the ''License'' node with the ''language'' attribute and the elements ''Text'' and ''URI''. The latter should contain a link to a well-defined license, for example https://creativecommons.org/publicdomain/zero/1.0/. More details on GBIF licensing can be found [http://www.gbif.org/terms/licences here]. ==Specimen Data== In ACBD terminology, a single specimen/observation is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a coontrolled vocabulary. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="9" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |ScientificName |FullScientificNameString* |Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |Complete name of the taxon including Authors (and years for animals). |- |rowspan="5" | ScientificName/NameAtomised/Botanical |GenusOrMonomial |Aaronsohnia |rowspan="5" | ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |- |FirstEpithet |factorovskyi |- |InfraspecificEpithet |factorovskyi |- |Rank |var. |- |AuthorTeam |Warb. & Eig. |- | - |NameComments | |Free text comments related to the name identified for the unit. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===Multimedia=== For providing links to associated images, videos and sounds, the sub-tree ''MultimediaObjects'' can be used. Each ABCD unit can have multiple multimedia objects, each represented by its own ''MultimediaObject'' with the following elements: {| class="wikitable" !ABCD element !Example !Remark |- |FileURI* |http://ww2.bgbm.org/herbarium/images/B/-W/14/61/B_-W_14616%20-00%200.jpg |The Universal Resource Identifier (URI) of the underlying multimedia file. |- |ProductURI |http://ww2.bgbm.org/herbarium/view_biocase.cfm?SpecimenPK=726 |The Universal Resource Identifier of an electronic source of a multimedia object including HTML, Javascript or similar means delivering rendered output. |- |Comment |Image of herbarium specimen E00732880 by Specimen Digitisation Pipeline. |Short description of the multimedia object. |- |Format |image/jpeg |A code or descriptive term for the file or object format. |- |FileSize |2365 |The size in Kb of the file referred to by the URI or filename for the multimedia object. |- |Creator |Paris, Museum National d'Histoire Naturelle, OD |The identifier (e.g. name or other id) of personnel who has taken the image. |- |CreatedDate |2012-07-07 |The date the multimedia object was created. |- |ID |22658 |Identifier for the multimedia object relating to or forming the unit. This may be a filename. |- |IPR | |Intellectual Property Rights relating specifically to the multimedia object. This subtree has the same elements as ''Metadata/IPRStatements''. Fore more details, see the respective [[ABCD2Concepts#General_metadata | remarks]]. |- |} a1cc2021379790b94998ba5642fb70511db2400a 1063 1062 2018-10-01T08:51:16Z JoergHoletschek 3 /* General unit data */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens/observations. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact* |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact* |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData* |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation* |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} '''Note:''' If you publish your data to GBIF, you are required to specify a license in the ''License'' node with the ''language'' attribute and the elements ''Text'' and ''URI''. The latter should contain a link to a well-defined license, for example https://creativecommons.org/publicdomain/zero/1.0/. More details on GBIF licensing can be found [http://www.gbif.org/terms/licences here]. ==Specimen Data== In ACBD terminology, a single specimen/observation is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |PreservedSpecimen |An indication of what the unit record describes; needs to comply with a [http://tdwg.github.io/dwc/terms/index.htm#basisOfRecord controlled vocabulary]. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="9" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |ScientificName |FullScientificNameString* |Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |Complete name of the taxon including Authors (and years for animals). |- |rowspan="5" | ScientificName/NameAtomised/Botanical |GenusOrMonomial |Aaronsohnia |rowspan="5" | ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |- |FirstEpithet |factorovskyi |- |InfraspecificEpithet |factorovskyi |- |Rank |var. |- |AuthorTeam |Warb. & Eig. |- | - |NameComments | |Free text comments related to the name identified for the unit. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===Multimedia=== For providing links to associated images, videos and sounds, the sub-tree ''MultimediaObjects'' can be used. Each ABCD unit can have multiple multimedia objects, each represented by its own ''MultimediaObject'' with the following elements: {| class="wikitable" !ABCD element !Example !Remark |- |FileURI* |http://ww2.bgbm.org/herbarium/images/B/-W/14/61/B_-W_14616%20-00%200.jpg |The Universal Resource Identifier (URI) of the underlying multimedia file. |- |ProductURI |http://ww2.bgbm.org/herbarium/view_biocase.cfm?SpecimenPK=726 |The Universal Resource Identifier of an electronic source of a multimedia object including HTML, Javascript or similar means delivering rendered output. |- |Comment |Image of herbarium specimen E00732880 by Specimen Digitisation Pipeline. |Short description of the multimedia object. |- |Format |image/jpeg |A code or descriptive term for the file or object format. |- |FileSize |2365 |The size in Kb of the file referred to by the URI or filename for the multimedia object. |- |Creator |Paris, Museum National d'Histoire Naturelle, OD |The identifier (e.g. name or other id) of personnel who has taken the image. |- |CreatedDate |2012-07-07 |The date the multimedia object was created. |- |ID |22658 |Identifier for the multimedia object relating to or forming the unit. This may be a filename. |- |IPR | |Intellectual Property Rights relating specifically to the multimedia object. This subtree has the same elements as ''Metadata/IPRStatements''. Fore more details, see the respective [[ABCD2Concepts#General_metadata | remarks]]. |- |} 877c2226289ff4d9761e487e9e533114f3fd31d8 0 22 1058 983 2018-10-01T08:26:52Z JoergHoletschek 3 /* Mandatory */ wikitext text/x-wiki This page lists the most commonly used ABCD 2.06 concepts. The ABCD documentation can be found in the [http://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki], a tabular list of ABCD concepts [http://terms.tdwg.org/wiki/ABCD_2_XPath_to_Short_Name_Mapping here]. == Mandatory == List of mandatory concepts. Without them being mapped, the Provider Software will not be able to create valid ABCD document: <pre> DataSets └─ DataSet ├─ ContentContacts │ └─ ContentContact │ └─ Name ├─ TechnicalContacts │ └─ TechnicalContact │ └─ Name ├─ Metadata │ ├─ Description │ │ └─ Representation │ │ ├─ @language │ │ └─ Title │ └─ RevisionData │ └─ DateModified └─ Units └─ Unit ├─ SourceInstitutionID ├─ SourceID └─ UnitID </pre> == Highly Recommended == List of recommended ABCD concepts. If you can provide this data, please do so, they're commonly used entry points to your data. '''Note:''' If you publish your data to GBIF, you are required to specify a license in the ''License'' node with the ''language'' attribute and the elements ''Text'' and ''URI''. The latter should contain a link to a well-defined license, for example https://creativecommons.org/publicdomain/zero/1.0/. More details on GBIF licensing can be found [http://www.gbif.org/terms/licences here]. <pre> DataSets └─ DataSet ├─ ContentContacts │ └─ ContentContact │ ├─ Email │ └─ Address ├─ TechnicalContacts │ └─ TechnicalContact │ ├─ Email │ └─ Address ├─ Metadata │ ├─ Description │ │ └─ Representation │ │ └─ Details │ ├─ Owners │ │ └─ Owner │ │ ├─ LogoURI │ │ ├─ URIs │ │ │ └─ URL │ │ ├─ Person │ │ │ └─ FullName │ │ └─ Organisation │ │ └─ Name │ │ └─ Representation │ │ └─ Text │ └─ IPRStatements │ └─ Licenses │ └─ License │ ├─ @language │ ├─ Text │ └─ URI └─ Units └─ Unit ├─ Identifications │ └─ Identification │ └─ Result │ ├─ @PreferredFlag │ ├─ TaxonIdentified │ │ └─ ScientificName │ │ └─ FullScientificNameString │ └─ HigherTaxa │ └─ HigherTaxon │ ├─ HigherTaxonName │ └─ HigherTaxonRank ├─ RecordBasis └─ Gathering ├─ DateTime │ └─ ISODateTimeBegin ├─ Agents │ └─ GatheringAgent │ └─ Person │ └─ FullName ├─ LocalityText ├─ Country │ ├─ ISO3166Code │ └─ Name └─ SiteCoordinateSets └─ SiteCoordinates └─ CoordinatesLatLong ├─ LongitudeDecimal ├─ LatitudeDecimal └─ CoordinateErrorDistanceInMeters </pre> == Commonly Used == List of further commonly mapped ABCD concepts. <pre> DataSets └─ DataSet ├─ Metadata │ └─ IPRStatements │ ├─ TermsOfUseStatements │ │ └─ TermsOfUse │ │ └─ Text │ ├─ Copyrights │ │ └─ Copyright │ │ └─ Text │ ├─ IPRDeclarations │ │ └─ IPRDeclaration │ │ └─ Text │ ├─ Licenses │ │ └─ License │ │ └─ Text │ ├─ Disclaimers │ │ └─ Disclaimer │ │ └─ Text │ ├─ Citations │ │ └─ Citation │ │ └─ Text │ └─ Acknowledgements │ └─ Acknowledgement │ └─ Text └─ Units └─ Unit ├─ DateLastEdited ├─ Identifications │ └─ Identification │ ├─ Date │ │ └─ ISODateTimeBegin │ └─ Identifiers │ └─ Identifier │ └─ IdentifierPersonName │ └─ FullName ├─ KindOfUnit ├─ CollectorsFieldNumber ├─ Sex ├─ Age ├─ Notes ├─ RecordURI ├─ MultiMediaObjects │ └─ MultiMediaObject │ ├─ FileURI │ ├─ Comment │ └─ Context ├─ SpecimenUnit │ └─ NomenclaturalTypeDesignations │ └─ NomenclaturalTypeDesignation │ ├─ TypeStatus │ ├─ TypifiedName │ │ └─ FullScientificNameString │ └─ NomenclaturalReference │ ├─ TitleCitation │ └─ CitationDetail └─ Gathering ├─ Project │ └─ ProjectTitle ├─ Biotope │ └─ Name ├─ Altitude │ └─ MeasurementOrFactAtomised │ ├─ LowerValue │ └─ UnitOfMeasurement ├─ Depth │ └─ MeasurementOrFactAtomised │ ├─ LowerValue │ └─ UnitOfMeasurement └─ NamedAreas └─ NamedArea ├─ AreaName └─ AreaClass </pre> d3b5cdcfe3619c2a9a77f03f74823e295529329b 1064 1058 2018-10-01T08:53:43Z JoergHoletschek 3 wikitext text/x-wiki This page lists the mandatory and most commonly used ABCD 2.06 concepts. For examples on the differenct concepts and their short explanations see [[ABCD2Concepts]]. The full ABCD documentation can be found in the [http://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki], a tabular list of ABCD concepts [http://terms.tdwg.org/wiki/ABCD_2_XPath_to_Short_Name_Mapping here]. == Mandatory == List of mandatory concepts. Without them being mapped, the Provider Software will not be able to create valid ABCD document: <pre> DataSets └─ DataSet ├─ ContentContacts │ └─ ContentContact │ └─ Name ├─ TechnicalContacts │ └─ TechnicalContact │ └─ Name ├─ Metadata │ ├─ Description │ │ └─ Representation │ │ ├─ @language │ │ └─ Title │ └─ RevisionData │ └─ DateModified └─ Units └─ Unit ├─ SourceInstitutionID ├─ SourceID └─ UnitID </pre> == Highly Recommended == List of recommended ABCD concepts. If you can provide this data, please do so, they're commonly used entry points to your data. '''Note:''' If you publish your data to GBIF, you are required to specify a license in the ''License'' node with the ''language'' attribute and the elements ''Text'' and ''URI''. The latter should contain a link to a well-defined license, for example https://creativecommons.org/publicdomain/zero/1.0/. More details on GBIF licensing can be found [http://www.gbif.org/terms/licences here]. <pre> DataSets └─ DataSet ├─ ContentContacts │ └─ ContentContact │ ├─ Email │ └─ Address ├─ TechnicalContacts │ └─ TechnicalContact │ ├─ Email │ └─ Address ├─ Metadata │ ├─ Description │ │ └─ Representation │ │ └─ Details │ ├─ Owners │ │ └─ Owner │ │ ├─ LogoURI │ │ ├─ URIs │ │ │ └─ URL │ │ ├─ Person │ │ │ └─ FullName │ │ └─ Organisation │ │ └─ Name │ │ └─ Representation │ │ └─ Text │ └─ IPRStatements │ └─ Licenses │ └─ License │ ├─ @language │ ├─ Text │ └─ URI └─ Units └─ Unit ├─ Identifications │ └─ Identification │ └─ Result │ ├─ @PreferredFlag │ ├─ TaxonIdentified │ │ └─ ScientificName │ │ └─ FullScientificNameString │ └─ HigherTaxa │ └─ HigherTaxon │ ├─ HigherTaxonName │ └─ HigherTaxonRank ├─ RecordBasis └─ Gathering ├─ DateTime │ └─ ISODateTimeBegin ├─ Agents │ └─ GatheringAgent │ └─ Person │ └─ FullName ├─ LocalityText ├─ Country │ ├─ ISO3166Code │ └─ Name └─ SiteCoordinateSets └─ SiteCoordinates └─ CoordinatesLatLong ├─ LongitudeDecimal ├─ LatitudeDecimal └─ CoordinateErrorDistanceInMeters </pre> == Commonly Used == List of further commonly mapped ABCD concepts. <pre> DataSets └─ DataSet ├─ Metadata │ └─ IPRStatements │ ├─ TermsOfUseStatements │ │ └─ TermsOfUse │ │ └─ Text │ ├─ Copyrights │ │ └─ Copyright │ │ └─ Text │ ├─ IPRDeclarations │ │ └─ IPRDeclaration │ │ └─ Text │ ├─ Licenses │ │ └─ License │ │ └─ Text │ ├─ Disclaimers │ │ └─ Disclaimer │ │ └─ Text │ ├─ Citations │ │ └─ Citation │ │ └─ Text │ └─ Acknowledgements │ └─ Acknowledgement │ └─ Text └─ Units └─ Unit ├─ DateLastEdited ├─ Identifications │ └─ Identification │ ├─ Date │ │ └─ ISODateTimeBegin │ └─ Identifiers │ └─ Identifier │ └─ IdentifierPersonName │ └─ FullName ├─ KindOfUnit ├─ CollectorsFieldNumber ├─ Sex ├─ Age ├─ Notes ├─ RecordURI ├─ MultiMediaObjects │ └─ MultiMediaObject │ ├─ FileURI │ ├─ Comment │ └─ Context ├─ SpecimenUnit │ └─ NomenclaturalTypeDesignations │ └─ NomenclaturalTypeDesignation │ ├─ TypeStatus │ ├─ TypifiedName │ │ └─ FullScientificNameString │ └─ NomenclaturalReference │ ├─ TitleCitation │ └─ CitationDetail └─ Gathering ├─ Project │ └─ ProjectTitle ├─ Biotope │ └─ Name ├─ Altitude │ └─ MeasurementOrFactAtomised │ ├─ LowerValue │ └─ UnitOfMeasurement ├─ Depth │ └─ MeasurementOrFactAtomised │ ├─ LowerValue │ └─ UnitOfMeasurement └─ NamedAreas └─ NamedArea ├─ AreaName └─ AreaClass </pre> 49b36d8a3a29d12cf96213eb2175f0d930b7d4d6 0 29 1065 900 2018-10-01T08:54:37Z JoergHoletschek 3 /* Access to Biological Collection Data Schema (ABCD) */ wikitext text/x-wiki This tutorial will explain how to set up a mapping for the ABCD2.06 data schema. If you intend to map another schema (e.g. one of the ABCD extension ABCD-EFG, ABCD-DNA or HISPID), you still should take the time to read it. Large portions of the process will be identical. == Access to Biological Collection Data Schema (ABCD) == ABCD in its current version 2.06 is the data schema typically used in conjunction with BioCASe. It is a highly complex XML data schema with about 1,000 elements that is able to store almost every piece of information that can be found in a natural history specimen collection or an observation database. The complexity is due to the fact that it can be used for a wide range of collections/databases – for living and preserved specimens, observations and culture collections, for zoological, botanical, bacterial and viral collections, marine or terrestrial, for herbaria, botanic and zoological gardens. For each of these special types, ABCD features special sections where information specific for this type can be stored. Other sections of ABCD will be shared by all these types, for example for gathering site, identifier or metadata. The full ABCD documentation can be found in the http://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki], a list of commonly used ABCD elements on the Wiki page [[CommonABCD2Concepts|Common ABCD2 Concepts]] and a comprehensive list of example values on the page [[ABCD2Concepts]]. == Creating an ABCD Mapping for a Data Source == If you created a data source from a template (e.g. ''abcdmetadata''), a schema mapping for ABCD will be already existent. In this case, you can directly jump to the next section. If you created an empty data source, the section <small>Schemas</small> in the data source configuration overview will be empty: [[File:abcdSchemaMappings.png|border]] In order to create a new schema mapping, select the desired schema from the list and click <small>Create</small>. For ABCD, choose <small>ABCD_2.06.xml</small> (be careful not to hit one of the ABCD extensions, they look quite similar). After clicking <small>Create</small>, you will be directed to the mapping editor, which will allow you to edit the mappings. == The Mapping Editor == The mapping editor allows you to add, remove and edit ABCD mappings. On top you’ll find some summary information, on the bottom you’ll see the structure of the ABCD mapping: [[File:abcdMappingEditorEmpty.png|border]] First thing we will do is to set the root table, which is the table that holds the records you want to publish (specimens or observations). That table must hold one record per object to be published, identified by a unique identifier (called ''UnitID''). This identifier doesn't need to be the primary key necessarily, as long as it is unique (if that sentence confuses you, ignore it and just keep in mind that the UnitID must be unique.) Use the drop down box labelled <small>Root table Alias</small> to select the root table of your data model. If your metadata table holds just one record (because all occurrences published share the same metadata) and if it is not linked to the root table by a foreign key, you can specify this table in the drop-down box <small>Static table Alias</small>. The Provider Software will use a natural join to connect the metadata record stored in this table to all occurrences published. If your metadata table is referenced by a foreign key in the root able, you must not declare the metadata table as a static table! So either join the metadata table using a foreign key OR declare it as a static table. '''Note:''' If you're using Access, you cannot use a static table, because Access doesn't support that. So when using Access, you always have to reference the metadata table with a foreign key. When you’ve set the root table (and static table, if you use that), press <small>Save</small>. As with the DB structure setup, this will write your changes to the configuration files, closing the tab without saving will discard your changes! Also, in case you messed something up, you can use the <small>Revert</small> button to restore the state when you’ve saved for the last time. In the mapping editor, the link <small>Overview</small> at the top of the page will take you back to the datasource configuration overview. After creating a mapping, it will appear in the section <small>Schemas</small>. The entry will show the number of mapped elements (0 for now) and the schema namespace. Pressing the trash can symbol will move it to the trash can, which means that your web service does not support this schema anymore; if you did this by mistake, you can restore it by pressing the <small>Restore</small> button. <small>Purge</small> will delete the schema mapping permanently: [[File:abcdSchemaMappings2.png|border]] You can map several schemas for a given datasource. For example, in addition to the current ABCD2 standard, you could add the deprecated version 1.2 (even though I have no idea why you should do that) or DarwinCore. Just select the appropriate schema in the drop-down list and click <small>Create</small>. Since the namespace of a schema will be used in requests, you cannot create two mappings for the same namespace. Trying this will get you an error message. == Adding Mandatory ABCD Elements == Clicking on a schema in the datasource overview page will take you again to the mapping editor. In the lower half of the page, you will see the basic structure of an ABCD document: [[File:abcdMappingEditorEmptyBottom.png|border]] <strong>Technical/Content Contact:</strong> These trees hold information on who should be contacted for technical issues or questions concerning the published database, each with name, address, email address and telephone number. <strong>Metadata:</strong> In this sub tree you’ll find all metadata fields, for example title and description of the dataset, ownership information, date last modified, IPR statements etc. <strong>Units:</strong> This part stores the "real" data you want to publish, that means all specimen or observation data. The three elements already shown in the tree are identifiers that make up the so called "Triple ID", which uniquely identifies records within primary biodiversity networks such as BioCASe and GBIF: * ''UnitID'', which is a unique identifier for each record to be published (you can use the catalogue number, for example, or an existing barcode); * ''SourceInstitutionID'', an identifier for the institution publishing the dataset (for example BGBM); * ''SourceID'', an identifier for the dataset (e.g. “Herbarium Berolinense’’). The elements already shown in the mapping editor are printed in red because they are mandatory, which means they are required for an ABCD document. Without them being mapped (to database fields that are non-empty), the Provider Software will not be able to construct valid ABCD documents and will not publish any records. So the first elements that should be mapped are the mandatory fields. To add a mapping, click the <strong>+</strong> symbol next to the field, which will open the mapping editor dialog, where you can select the database table/column you want to map the ABCD element to: [[File:abcdMappingEditorDialog.png|border]] In this case, the element ''/DataSets/DataSet/Metadata/RevisionData/DateModified'' is mapped to the column ''source_update'' of the ''metadata'' table, which has the data type ''date''. (Remember that table/column retrieval is not supported for all DBMS, so if you’re unlucky, you might need to type in table/column name manually.) <small>Save</small> will accept your settings and close the dialog, <small>Cancel</small> will discard any changes. Instead of mapping an element to a database field, you can map it to a literal. For example, you could enter the technical and content contact information directly as literals in the mapping editor dialog (assuming they’re the same for all records). However, we strongly discourage you from doing this. For one thing, this is just bad style, since it will scatter the information published over different places: Some will be retrieved from the database, some (the literals) will be loaded from the configuration files. Changing the metadata of your service later would then require access to the database and access to the BioCASe configuration – that means the configuration tool password. Be nice to the person who will do these changes and put everything neatly in a database table! The second reason is even more important: Using literals for an element will result in this element to become not searchable, which means it cannot be used in a filter of a Search or a Scan request. Even though this behaviour is discoverable with a Capabilities request (by evaluating the ''searchable'' flag), a potential user of your web service might just not think of that, so it’s a good idea to avoid that in first place. Please use literals sparsely! In the mapping editor window, you can select several source columns and several literals, even a combination of both. The Provider Software will simply concatenate the values retrieved from the database and the literals specified (in the order they’re given). But be careful: any use of concatenation or literals will make the element not searchable. A better solution is to do such manipulations earlier, in a database view, and direct the mapping editor to a single resulting database field. Use the mapping editor window to add mappings for all eight mandatory fields: Click on the <strong>+</strong> symbol next to the element, choose table/column name and the correct data type, then press <small>Save</small>. When you’re done, the page should look similar to the following image. Press <small>Save</small> in the main window to write your changes to the configuration files. [[File:abcdMappingEditorMandatoryElements.png|border]] == Initial Testing of the Data Source == Once you’ve mapped all mandatory elements, you can do the first test of your BioCASe web service. Even though you haven’t mapped any real information yet, it is a good idea to check the service and solve any problems hitherto before continuing with the mapping. Click on <small>Test mapping!</small> to open the Query Form that can be used for this purpose: [[File:abcdQueryFormEmpty.png|border]] With the Query Form you can send a BioCASe request to your web service; it will wait for the reply of the web service and display the response document. On top of the page you can see the URL of your web service. In this case it is ''<nowiki>http://localhost/biocase/pywrapper.cgi?dsa=flora</nowiki>'', ''flora'' being the name of the datasource. Make sure this is correct for your installation! Into the text box below you can enter the request XML text. Because it would be cumbersome to enter the whole BioCASe request, you can use the links below the box to load templates. For now, click on <small>ABCD2 Search</small> to fill the box with a BioCASe Search request for documents in the ABCD2 format (the one you’ve just mapped): <syntaxhighlight> <?xml version='1.0' encoding='UTF-8'?> <request xmlns='http://www.biocase.org/schemas/protocol/1.3'> <header><type>search</type></header> <search> <requestFormat>http://www.tdwg.org/schemas/abcd/2.06</requestFormat> <responseFormat start='0' limit='10'>http://www.tdwg.org/schemas/abcd/2.06</responseFormat> <filter> <like path='/DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString'>A*</like> </filter> <count>false</count> </search> </request> </syntaxhighlight> As you can see, there is a filter set on the ''ScientificName'' element. Since we haven’t mapped this element yet, this won’t work. So let’s remove the filter (all lines between <filter> and </filter>, including the ''filter'' tags), leaving the text box like this: [[File:abcdQueryFormFilled.png|border]] Pressing <small>Submit</small> will send the request to the web service and – hopefully – display the response document with the correct ABCD records. For the mapping we’ve just created, it should look similar to the following (use the "–" symbol next to the trees TechnicalContact, ContentContact and Metadata to collapse them): [[File:abcdResponseMandatoryElements.png|border]] If you get an error, please check out the [[Debugging]] tutorial on how to debug a web service. You should continue the mapping process only after your web service returns results for the basal mapping we’ve just created. == Adding additional ABCD Elements == Once you’ve successfully managed to map the mandatory ABCD elements ("successfully" means the web service returns ABCD documents), you can go on with the mapping process. This is usually an iterative process: You add a handful of elements (for example, country, country code, locality text and coordinates for the gathering site), then you use the Query Form to see if the information ends up where it is supposed to do. This is the reason the Query Form opens up in a separate tab (or window, of you configured your web browser so): In one tab, you use the mapping editor to add new elements; after pressing <small>Save</small> you go to the Query Form tab and see what the response documents look like. Redo this until you’ve mapped all information you want to publish. In order to map new ABCD concepts, tick the check box labelled <small>Show all concepts</small> and press <small>Refresh</small>. The mapping editor will then display the whole ABCD tree, which looks admittedly a bit intimidating. In order to find the concept you’re looking for, use the search function of your browser. If you want to map the country name of the gathering site, for example, hit <small>Ctrl + F</small> (for Firefox, Microsoft IE and Opera) and type in the name of the ABCD element, namely ''country''. The browser will jump to the ''Country'' tree of the ''Gathering'' element, where you’ll find the elements for country name (''Name''), ISO country code (''ISO3166Code'') and the country name in the country’s language (''NameDerived''): [[File:abcdMappingEditorSearch.png|500px]] Continue the sequence mapping new elements/testing until you’ve mapped all information you want to publish. You can have a look at the list of recommend and commonly used ABCD elements on the [[CommonABCD2Concepts|Common ABCD2 Concepts]] page. The full ABCD documentation can be found in the [http://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. The following page will show you a sample response ABCD document for a web service with 77 concepts mapped. Of course your documents will look different, because you will probably use different ABCD elements, but still it’s good to get an idea: [[SampleABCDDocument]]. f576b6f7dd297fb25f2fb1d410c75ce6b4377054 1066 1065 2018-10-01T08:56:40Z JoergHoletschek 3 /* Adding additional ABCD Elements */ wikitext text/x-wiki This tutorial will explain how to set up a mapping for the ABCD2.06 data schema. If you intend to map another schema (e.g. one of the ABCD extension ABCD-EFG, ABCD-DNA or HISPID), you still should take the time to read it. Large portions of the process will be identical. == Access to Biological Collection Data Schema (ABCD) == ABCD in its current version 2.06 is the data schema typically used in conjunction with BioCASe. It is a highly complex XML data schema with about 1,000 elements that is able to store almost every piece of information that can be found in a natural history specimen collection or an observation database. The complexity is due to the fact that it can be used for a wide range of collections/databases – for living and preserved specimens, observations and culture collections, for zoological, botanical, bacterial and viral collections, marine or terrestrial, for herbaria, botanic and zoological gardens. For each of these special types, ABCD features special sections where information specific for this type can be stored. Other sections of ABCD will be shared by all these types, for example for gathering site, identifier or metadata. The full ABCD documentation can be found in the http://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki], a list of commonly used ABCD elements on the Wiki page [[CommonABCD2Concepts|Common ABCD2 Concepts]] and a comprehensive list of example values on the page [[ABCD2Concepts]]. == Creating an ABCD Mapping for a Data Source == If you created a data source from a template (e.g. ''abcdmetadata''), a schema mapping for ABCD will be already existent. In this case, you can directly jump to the next section. If you created an empty data source, the section <small>Schemas</small> in the data source configuration overview will be empty: [[File:abcdSchemaMappings.png|border]] In order to create a new schema mapping, select the desired schema from the list and click <small>Create</small>. For ABCD, choose <small>ABCD_2.06.xml</small> (be careful not to hit one of the ABCD extensions, they look quite similar). After clicking <small>Create</small>, you will be directed to the mapping editor, which will allow you to edit the mappings. == The Mapping Editor == The mapping editor allows you to add, remove and edit ABCD mappings. On top you’ll find some summary information, on the bottom you’ll see the structure of the ABCD mapping: [[File:abcdMappingEditorEmpty.png|border]] First thing we will do is to set the root table, which is the table that holds the records you want to publish (specimens or observations). That table must hold one record per object to be published, identified by a unique identifier (called ''UnitID''). This identifier doesn't need to be the primary key necessarily, as long as it is unique (if that sentence confuses you, ignore it and just keep in mind that the UnitID must be unique.) Use the drop down box labelled <small>Root table Alias</small> to select the root table of your data model. If your metadata table holds just one record (because all occurrences published share the same metadata) and if it is not linked to the root table by a foreign key, you can specify this table in the drop-down box <small>Static table Alias</small>. The Provider Software will use a natural join to connect the metadata record stored in this table to all occurrences published. If your metadata table is referenced by a foreign key in the root able, you must not declare the metadata table as a static table! So either join the metadata table using a foreign key OR declare it as a static table. '''Note:''' If you're using Access, you cannot use a static table, because Access doesn't support that. So when using Access, you always have to reference the metadata table with a foreign key. When you’ve set the root table (and static table, if you use that), press <small>Save</small>. As with the DB structure setup, this will write your changes to the configuration files, closing the tab without saving will discard your changes! Also, in case you messed something up, you can use the <small>Revert</small> button to restore the state when you’ve saved for the last time. In the mapping editor, the link <small>Overview</small> at the top of the page will take you back to the datasource configuration overview. After creating a mapping, it will appear in the section <small>Schemas</small>. The entry will show the number of mapped elements (0 for now) and the schema namespace. Pressing the trash can symbol will move it to the trash can, which means that your web service does not support this schema anymore; if you did this by mistake, you can restore it by pressing the <small>Restore</small> button. <small>Purge</small> will delete the schema mapping permanently: [[File:abcdSchemaMappings2.png|border]] You can map several schemas for a given datasource. For example, in addition to the current ABCD2 standard, you could add the deprecated version 1.2 (even though I have no idea why you should do that) or DarwinCore. Just select the appropriate schema in the drop-down list and click <small>Create</small>. Since the namespace of a schema will be used in requests, you cannot create two mappings for the same namespace. Trying this will get you an error message. == Adding Mandatory ABCD Elements == Clicking on a schema in the datasource overview page will take you again to the mapping editor. In the lower half of the page, you will see the basic structure of an ABCD document: [[File:abcdMappingEditorEmptyBottom.png|border]] <strong>Technical/Content Contact:</strong> These trees hold information on who should be contacted for technical issues or questions concerning the published database, each with name, address, email address and telephone number. <strong>Metadata:</strong> In this sub tree you’ll find all metadata fields, for example title and description of the dataset, ownership information, date last modified, IPR statements etc. <strong>Units:</strong> This part stores the "real" data you want to publish, that means all specimen or observation data. The three elements already shown in the tree are identifiers that make up the so called "Triple ID", which uniquely identifies records within primary biodiversity networks such as BioCASe and GBIF: * ''UnitID'', which is a unique identifier for each record to be published (you can use the catalogue number, for example, or an existing barcode); * ''SourceInstitutionID'', an identifier for the institution publishing the dataset (for example BGBM); * ''SourceID'', an identifier for the dataset (e.g. “Herbarium Berolinense’’). The elements already shown in the mapping editor are printed in red because they are mandatory, which means they are required for an ABCD document. Without them being mapped (to database fields that are non-empty), the Provider Software will not be able to construct valid ABCD documents and will not publish any records. So the first elements that should be mapped are the mandatory fields. To add a mapping, click the <strong>+</strong> symbol next to the field, which will open the mapping editor dialog, where you can select the database table/column you want to map the ABCD element to: [[File:abcdMappingEditorDialog.png|border]] In this case, the element ''/DataSets/DataSet/Metadata/RevisionData/DateModified'' is mapped to the column ''source_update'' of the ''metadata'' table, which has the data type ''date''. (Remember that table/column retrieval is not supported for all DBMS, so if you’re unlucky, you might need to type in table/column name manually.) <small>Save</small> will accept your settings and close the dialog, <small>Cancel</small> will discard any changes. Instead of mapping an element to a database field, you can map it to a literal. For example, you could enter the technical and content contact information directly as literals in the mapping editor dialog (assuming they’re the same for all records). However, we strongly discourage you from doing this. For one thing, this is just bad style, since it will scatter the information published over different places: Some will be retrieved from the database, some (the literals) will be loaded from the configuration files. Changing the metadata of your service later would then require access to the database and access to the BioCASe configuration – that means the configuration tool password. Be nice to the person who will do these changes and put everything neatly in a database table! The second reason is even more important: Using literals for an element will result in this element to become not searchable, which means it cannot be used in a filter of a Search or a Scan request. Even though this behaviour is discoverable with a Capabilities request (by evaluating the ''searchable'' flag), a potential user of your web service might just not think of that, so it’s a good idea to avoid that in first place. Please use literals sparsely! In the mapping editor window, you can select several source columns and several literals, even a combination of both. The Provider Software will simply concatenate the values retrieved from the database and the literals specified (in the order they’re given). But be careful: any use of concatenation or literals will make the element not searchable. A better solution is to do such manipulations earlier, in a database view, and direct the mapping editor to a single resulting database field. Use the mapping editor window to add mappings for all eight mandatory fields: Click on the <strong>+</strong> symbol next to the element, choose table/column name and the correct data type, then press <small>Save</small>. When you’re done, the page should look similar to the following image. Press <small>Save</small> in the main window to write your changes to the configuration files. [[File:abcdMappingEditorMandatoryElements.png|border]] == Initial Testing of the Data Source == Once you’ve mapped all mandatory elements, you can do the first test of your BioCASe web service. Even though you haven’t mapped any real information yet, it is a good idea to check the service and solve any problems hitherto before continuing with the mapping. Click on <small>Test mapping!</small> to open the Query Form that can be used for this purpose: [[File:abcdQueryFormEmpty.png|border]] With the Query Form you can send a BioCASe request to your web service; it will wait for the reply of the web service and display the response document. On top of the page you can see the URL of your web service. In this case it is ''<nowiki>http://localhost/biocase/pywrapper.cgi?dsa=flora</nowiki>'', ''flora'' being the name of the datasource. Make sure this is correct for your installation! Into the text box below you can enter the request XML text. Because it would be cumbersome to enter the whole BioCASe request, you can use the links below the box to load templates. For now, click on <small>ABCD2 Search</small> to fill the box with a BioCASe Search request for documents in the ABCD2 format (the one you’ve just mapped): <syntaxhighlight> <?xml version='1.0' encoding='UTF-8'?> <request xmlns='http://www.biocase.org/schemas/protocol/1.3'> <header><type>search</type></header> <search> <requestFormat>http://www.tdwg.org/schemas/abcd/2.06</requestFormat> <responseFormat start='0' limit='10'>http://www.tdwg.org/schemas/abcd/2.06</responseFormat> <filter> <like path='/DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString'>A*</like> </filter> <count>false</count> </search> </request> </syntaxhighlight> As you can see, there is a filter set on the ''ScientificName'' element. Since we haven’t mapped this element yet, this won’t work. So let’s remove the filter (all lines between <filter> and </filter>, including the ''filter'' tags), leaving the text box like this: [[File:abcdQueryFormFilled.png|border]] Pressing <small>Submit</small> will send the request to the web service and – hopefully – display the response document with the correct ABCD records. For the mapping we’ve just created, it should look similar to the following (use the "–" symbol next to the trees TechnicalContact, ContentContact and Metadata to collapse them): [[File:abcdResponseMandatoryElements.png|border]] If you get an error, please check out the [[Debugging]] tutorial on how to debug a web service. You should continue the mapping process only after your web service returns results for the basal mapping we’ve just created. == Adding additional ABCD Elements == Once you’ve successfully managed to map the mandatory ABCD elements ("successfully" means the web service returns ABCD documents), you can go on with the mapping process. This is usually an iterative process: You add a handful of elements (for example, country, country code, locality text and coordinates for the gathering site), then you use the Query Form to see if the information ends up where it is supposed to do. This is the reason the Query Form opens up in a separate tab (or window, of you configured your web browser so): In one tab, you use the mapping editor to add new elements; after pressing <small>Save</small> you go to the Query Form tab and see what the response documents look like. Redo this until you’ve mapped all information you want to publish. In order to map new ABCD concepts, tick the check box labelled <small>Show all concepts</small> and press <small>Refresh</small>. The mapping editor will then display the whole ABCD tree, which looks admittedly a bit intimidating. In order to find the concept you’re looking for, use the search function of your browser. If you want to map the country name of the gathering site, for example, hit <small>Ctrl + F</small> (for Firefox, Microsoft IE and Opera) and type in the name of the ABCD element, namely ''country''. The browser will jump to the ''Country'' tree of the ''Gathering'' element, where you’ll find the elements for country name (''Name''), ISO country code (''ISO3166Code'') and the country name in the country’s language (''NameDerived''): [[File:abcdMappingEditorSearch.png|500px]] Continue the sequence mapping new elements/testing until you’ve mapped all information you want to publish. You can have a look at the list of recommend and commonly used ABCD elements on the [[CommonABCD2Concepts|Common ABCD2 Concepts]] and [ABCD2Concepts] pages. The full ABCD documentation can be found in the [http://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. The following page will show you a sample response ABCD document for a web service with 77 concepts mapped. Of course your documents will look different, because you will probably use different ABCD elements, but still it’s good to get an idea: [[SampleABCDDocument]]. 24a85630c0a63af0fd9f602f19b6ae82482da502 1067 1066 2018-10-01T08:57:22Z JoergHoletschek 3 /* Adding additional ABCD Elements */ wikitext text/x-wiki This tutorial will explain how to set up a mapping for the ABCD2.06 data schema. If you intend to map another schema (e.g. one of the ABCD extension ABCD-EFG, ABCD-DNA or HISPID), you still should take the time to read it. Large portions of the process will be identical. == Access to Biological Collection Data Schema (ABCD) == ABCD in its current version 2.06 is the data schema typically used in conjunction with BioCASe. It is a highly complex XML data schema with about 1,000 elements that is able to store almost every piece of information that can be found in a natural history specimen collection or an observation database. The complexity is due to the fact that it can be used for a wide range of collections/databases – for living and preserved specimens, observations and culture collections, for zoological, botanical, bacterial and viral collections, marine or terrestrial, for herbaria, botanic and zoological gardens. For each of these special types, ABCD features special sections where information specific for this type can be stored. Other sections of ABCD will be shared by all these types, for example for gathering site, identifier or metadata. The full ABCD documentation can be found in the http://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki], a list of commonly used ABCD elements on the Wiki page [[CommonABCD2Concepts|Common ABCD2 Concepts]] and a comprehensive list of example values on the page [[ABCD2Concepts]]. == Creating an ABCD Mapping for a Data Source == If you created a data source from a template (e.g. ''abcdmetadata''), a schema mapping for ABCD will be already existent. In this case, you can directly jump to the next section. If you created an empty data source, the section <small>Schemas</small> in the data source configuration overview will be empty: [[File:abcdSchemaMappings.png|border]] In order to create a new schema mapping, select the desired schema from the list and click <small>Create</small>. For ABCD, choose <small>ABCD_2.06.xml</small> (be careful not to hit one of the ABCD extensions, they look quite similar). After clicking <small>Create</small>, you will be directed to the mapping editor, which will allow you to edit the mappings. == The Mapping Editor == The mapping editor allows you to add, remove and edit ABCD mappings. On top you’ll find some summary information, on the bottom you’ll see the structure of the ABCD mapping: [[File:abcdMappingEditorEmpty.png|border]] First thing we will do is to set the root table, which is the table that holds the records you want to publish (specimens or observations). That table must hold one record per object to be published, identified by a unique identifier (called ''UnitID''). This identifier doesn't need to be the primary key necessarily, as long as it is unique (if that sentence confuses you, ignore it and just keep in mind that the UnitID must be unique.) Use the drop down box labelled <small>Root table Alias</small> to select the root table of your data model. If your metadata table holds just one record (because all occurrences published share the same metadata) and if it is not linked to the root table by a foreign key, you can specify this table in the drop-down box <small>Static table Alias</small>. The Provider Software will use a natural join to connect the metadata record stored in this table to all occurrences published. If your metadata table is referenced by a foreign key in the root able, you must not declare the metadata table as a static table! So either join the metadata table using a foreign key OR declare it as a static table. '''Note:''' If you're using Access, you cannot use a static table, because Access doesn't support that. So when using Access, you always have to reference the metadata table with a foreign key. When you’ve set the root table (and static table, if you use that), press <small>Save</small>. As with the DB structure setup, this will write your changes to the configuration files, closing the tab without saving will discard your changes! Also, in case you messed something up, you can use the <small>Revert</small> button to restore the state when you’ve saved for the last time. In the mapping editor, the link <small>Overview</small> at the top of the page will take you back to the datasource configuration overview. After creating a mapping, it will appear in the section <small>Schemas</small>. The entry will show the number of mapped elements (0 for now) and the schema namespace. Pressing the trash can symbol will move it to the trash can, which means that your web service does not support this schema anymore; if you did this by mistake, you can restore it by pressing the <small>Restore</small> button. <small>Purge</small> will delete the schema mapping permanently: [[File:abcdSchemaMappings2.png|border]] You can map several schemas for a given datasource. For example, in addition to the current ABCD2 standard, you could add the deprecated version 1.2 (even though I have no idea why you should do that) or DarwinCore. Just select the appropriate schema in the drop-down list and click <small>Create</small>. Since the namespace of a schema will be used in requests, you cannot create two mappings for the same namespace. Trying this will get you an error message. == Adding Mandatory ABCD Elements == Clicking on a schema in the datasource overview page will take you again to the mapping editor. In the lower half of the page, you will see the basic structure of an ABCD document: [[File:abcdMappingEditorEmptyBottom.png|border]] <strong>Technical/Content Contact:</strong> These trees hold information on who should be contacted for technical issues or questions concerning the published database, each with name, address, email address and telephone number. <strong>Metadata:</strong> In this sub tree you’ll find all metadata fields, for example title and description of the dataset, ownership information, date last modified, IPR statements etc. <strong>Units:</strong> This part stores the "real" data you want to publish, that means all specimen or observation data. The three elements already shown in the tree are identifiers that make up the so called "Triple ID", which uniquely identifies records within primary biodiversity networks such as BioCASe and GBIF: * ''UnitID'', which is a unique identifier for each record to be published (you can use the catalogue number, for example, or an existing barcode); * ''SourceInstitutionID'', an identifier for the institution publishing the dataset (for example BGBM); * ''SourceID'', an identifier for the dataset (e.g. “Herbarium Berolinense’’). The elements already shown in the mapping editor are printed in red because they are mandatory, which means they are required for an ABCD document. Without them being mapped (to database fields that are non-empty), the Provider Software will not be able to construct valid ABCD documents and will not publish any records. So the first elements that should be mapped are the mandatory fields. To add a mapping, click the <strong>+</strong> symbol next to the field, which will open the mapping editor dialog, where you can select the database table/column you want to map the ABCD element to: [[File:abcdMappingEditorDialog.png|border]] In this case, the element ''/DataSets/DataSet/Metadata/RevisionData/DateModified'' is mapped to the column ''source_update'' of the ''metadata'' table, which has the data type ''date''. (Remember that table/column retrieval is not supported for all DBMS, so if you’re unlucky, you might need to type in table/column name manually.) <small>Save</small> will accept your settings and close the dialog, <small>Cancel</small> will discard any changes. Instead of mapping an element to a database field, you can map it to a literal. For example, you could enter the technical and content contact information directly as literals in the mapping editor dialog (assuming they’re the same for all records). However, we strongly discourage you from doing this. For one thing, this is just bad style, since it will scatter the information published over different places: Some will be retrieved from the database, some (the literals) will be loaded from the configuration files. Changing the metadata of your service later would then require access to the database and access to the BioCASe configuration – that means the configuration tool password. Be nice to the person who will do these changes and put everything neatly in a database table! The second reason is even more important: Using literals for an element will result in this element to become not searchable, which means it cannot be used in a filter of a Search or a Scan request. Even though this behaviour is discoverable with a Capabilities request (by evaluating the ''searchable'' flag), a potential user of your web service might just not think of that, so it’s a good idea to avoid that in first place. Please use literals sparsely! In the mapping editor window, you can select several source columns and several literals, even a combination of both. The Provider Software will simply concatenate the values retrieved from the database and the literals specified (in the order they’re given). But be careful: any use of concatenation or literals will make the element not searchable. A better solution is to do such manipulations earlier, in a database view, and direct the mapping editor to a single resulting database field. Use the mapping editor window to add mappings for all eight mandatory fields: Click on the <strong>+</strong> symbol next to the element, choose table/column name and the correct data type, then press <small>Save</small>. When you’re done, the page should look similar to the following image. Press <small>Save</small> in the main window to write your changes to the configuration files. [[File:abcdMappingEditorMandatoryElements.png|border]] == Initial Testing of the Data Source == Once you’ve mapped all mandatory elements, you can do the first test of your BioCASe web service. Even though you haven’t mapped any real information yet, it is a good idea to check the service and solve any problems hitherto before continuing with the mapping. Click on <small>Test mapping!</small> to open the Query Form that can be used for this purpose: [[File:abcdQueryFormEmpty.png|border]] With the Query Form you can send a BioCASe request to your web service; it will wait for the reply of the web service and display the response document. On top of the page you can see the URL of your web service. In this case it is ''<nowiki>http://localhost/biocase/pywrapper.cgi?dsa=flora</nowiki>'', ''flora'' being the name of the datasource. Make sure this is correct for your installation! Into the text box below you can enter the request XML text. Because it would be cumbersome to enter the whole BioCASe request, you can use the links below the box to load templates. For now, click on <small>ABCD2 Search</small> to fill the box with a BioCASe Search request for documents in the ABCD2 format (the one you’ve just mapped): <syntaxhighlight> <?xml version='1.0' encoding='UTF-8'?> <request xmlns='http://www.biocase.org/schemas/protocol/1.3'> <header><type>search</type></header> <search> <requestFormat>http://www.tdwg.org/schemas/abcd/2.06</requestFormat> <responseFormat start='0' limit='10'>http://www.tdwg.org/schemas/abcd/2.06</responseFormat> <filter> <like path='/DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString'>A*</like> </filter> <count>false</count> </search> </request> </syntaxhighlight> As you can see, there is a filter set on the ''ScientificName'' element. Since we haven’t mapped this element yet, this won’t work. So let’s remove the filter (all lines between <filter> and </filter>, including the ''filter'' tags), leaving the text box like this: [[File:abcdQueryFormFilled.png|border]] Pressing <small>Submit</small> will send the request to the web service and – hopefully – display the response document with the correct ABCD records. For the mapping we’ve just created, it should look similar to the following (use the "–" symbol next to the trees TechnicalContact, ContentContact and Metadata to collapse them): [[File:abcdResponseMandatoryElements.png|border]] If you get an error, please check out the [[Debugging]] tutorial on how to debug a web service. You should continue the mapping process only after your web service returns results for the basal mapping we’ve just created. == Adding additional ABCD Elements == Once you’ve successfully managed to map the mandatory ABCD elements ("successfully" means the web service returns ABCD documents), you can go on with the mapping process. This is usually an iterative process: You add a handful of elements (for example, country, country code, locality text and coordinates for the gathering site), then you use the Query Form to see if the information ends up where it is supposed to do. This is the reason the Query Form opens up in a separate tab (or window, of you configured your web browser so): In one tab, you use the mapping editor to add new elements; after pressing <small>Save</small> you go to the Query Form tab and see what the response documents look like. Redo this until you’ve mapped all information you want to publish. In order to map new ABCD concepts, tick the check box labelled <small>Show all concepts</small> and press <small>Refresh</small>. The mapping editor will then display the whole ABCD tree, which looks admittedly a bit intimidating. In order to find the concept you’re looking for, use the search function of your browser. If you want to map the country name of the gathering site, for example, hit <small>Ctrl + F</small> (for Firefox, Microsoft IE and Opera) and type in the name of the ABCD element, namely ''country''. The browser will jump to the ''Country'' tree of the ''Gathering'' element, where you’ll find the elements for country name (''Name''), ISO country code (''ISO3166Code'') and the country name in the country’s language (''NameDerived''): [[File:abcdMappingEditorSearch.png|500px]] Continue the sequence mapping new elements/testing until you’ve mapped all information you want to publish. You can have a look at the list of recommend and commonly used ABCD elements on the [[CommonABCD2Concepts|Common ABCD2 Concepts]] and [[ABCD2Concepts]] pages. The full ABCD documentation can be found in the [http://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. The following page will show you a sample response ABCD document for a web service with 77 concepts mapped. Of course your documents will look different, because you will probably use different ABCD elements, but still it’s good to get an idea: [[SampleABCDDocument]]. 614ecc2296cb229ee364bc28eec6adc1af77d105 1068 1067 2018-10-01T09:03:14Z JoergHoletschek 3 /* Initial Testing of the Data Source */ wikitext text/x-wiki This tutorial will explain how to set up a mapping for the ABCD2.06 data schema. If you intend to map another schema (e.g. one of the ABCD extension ABCD-EFG, ABCD-DNA or HISPID), you still should take the time to read it. Large portions of the process will be identical. == Access to Biological Collection Data Schema (ABCD) == ABCD in its current version 2.06 is the data schema typically used in conjunction with BioCASe. It is a highly complex XML data schema with about 1,000 elements that is able to store almost every piece of information that can be found in a natural history specimen collection or an observation database. The complexity is due to the fact that it can be used for a wide range of collections/databases – for living and preserved specimens, observations and culture collections, for zoological, botanical, bacterial and viral collections, marine or terrestrial, for herbaria, botanic and zoological gardens. For each of these special types, ABCD features special sections where information specific for this type can be stored. Other sections of ABCD will be shared by all these types, for example for gathering site, identifier or metadata. The full ABCD documentation can be found in the http://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki], a list of commonly used ABCD elements on the Wiki page [[CommonABCD2Concepts|Common ABCD2 Concepts]] and a comprehensive list of example values on the page [[ABCD2Concepts]]. == Creating an ABCD Mapping for a Data Source == If you created a data source from a template (e.g. ''abcdmetadata''), a schema mapping for ABCD will be already existent. In this case, you can directly jump to the next section. If you created an empty data source, the section <small>Schemas</small> in the data source configuration overview will be empty: [[File:abcdSchemaMappings.png|border]] In order to create a new schema mapping, select the desired schema from the list and click <small>Create</small>. For ABCD, choose <small>ABCD_2.06.xml</small> (be careful not to hit one of the ABCD extensions, they look quite similar). After clicking <small>Create</small>, you will be directed to the mapping editor, which will allow you to edit the mappings. == The Mapping Editor == The mapping editor allows you to add, remove and edit ABCD mappings. On top you’ll find some summary information, on the bottom you’ll see the structure of the ABCD mapping: [[File:abcdMappingEditorEmpty.png|border]] First thing we will do is to set the root table, which is the table that holds the records you want to publish (specimens or observations). That table must hold one record per object to be published, identified by a unique identifier (called ''UnitID''). This identifier doesn't need to be the primary key necessarily, as long as it is unique (if that sentence confuses you, ignore it and just keep in mind that the UnitID must be unique.) Use the drop down box labelled <small>Root table Alias</small> to select the root table of your data model. If your metadata table holds just one record (because all occurrences published share the same metadata) and if it is not linked to the root table by a foreign key, you can specify this table in the drop-down box <small>Static table Alias</small>. The Provider Software will use a natural join to connect the metadata record stored in this table to all occurrences published. If your metadata table is referenced by a foreign key in the root able, you must not declare the metadata table as a static table! So either join the metadata table using a foreign key OR declare it as a static table. '''Note:''' If you're using Access, you cannot use a static table, because Access doesn't support that. So when using Access, you always have to reference the metadata table with a foreign key. When you’ve set the root table (and static table, if you use that), press <small>Save</small>. As with the DB structure setup, this will write your changes to the configuration files, closing the tab without saving will discard your changes! Also, in case you messed something up, you can use the <small>Revert</small> button to restore the state when you’ve saved for the last time. In the mapping editor, the link <small>Overview</small> at the top of the page will take you back to the datasource configuration overview. After creating a mapping, it will appear in the section <small>Schemas</small>. The entry will show the number of mapped elements (0 for now) and the schema namespace. Pressing the trash can symbol will move it to the trash can, which means that your web service does not support this schema anymore; if you did this by mistake, you can restore it by pressing the <small>Restore</small> button. <small>Purge</small> will delete the schema mapping permanently: [[File:abcdSchemaMappings2.png|border]] You can map several schemas for a given datasource. For example, in addition to the current ABCD2 standard, you could add the deprecated version 1.2 (even though I have no idea why you should do that) or DarwinCore. Just select the appropriate schema in the drop-down list and click <small>Create</small>. Since the namespace of a schema will be used in requests, you cannot create two mappings for the same namespace. Trying this will get you an error message. == Adding Mandatory ABCD Elements == Clicking on a schema in the datasource overview page will take you again to the mapping editor. In the lower half of the page, you will see the basic structure of an ABCD document: [[File:abcdMappingEditorEmptyBottom.png|border]] <strong>Technical/Content Contact:</strong> These trees hold information on who should be contacted for technical issues or questions concerning the published database, each with name, address, email address and telephone number. <strong>Metadata:</strong> In this sub tree you’ll find all metadata fields, for example title and description of the dataset, ownership information, date last modified, IPR statements etc. <strong>Units:</strong> This part stores the "real" data you want to publish, that means all specimen or observation data. The three elements already shown in the tree are identifiers that make up the so called "Triple ID", which uniquely identifies records within primary biodiversity networks such as BioCASe and GBIF: * ''UnitID'', which is a unique identifier for each record to be published (you can use the catalogue number, for example, or an existing barcode); * ''SourceInstitutionID'', an identifier for the institution publishing the dataset (for example BGBM); * ''SourceID'', an identifier for the dataset (e.g. “Herbarium Berolinense’’). The elements already shown in the mapping editor are printed in red because they are mandatory, which means they are required for an ABCD document. Without them being mapped (to database fields that are non-empty), the Provider Software will not be able to construct valid ABCD documents and will not publish any records. So the first elements that should be mapped are the mandatory fields. To add a mapping, click the <strong>+</strong> symbol next to the field, which will open the mapping editor dialog, where you can select the database table/column you want to map the ABCD element to: [[File:abcdMappingEditorDialog.png|border]] In this case, the element ''/DataSets/DataSet/Metadata/RevisionData/DateModified'' is mapped to the column ''source_update'' of the ''metadata'' table, which has the data type ''date''. (Remember that table/column retrieval is not supported for all DBMS, so if you’re unlucky, you might need to type in table/column name manually.) <small>Save</small> will accept your settings and close the dialog, <small>Cancel</small> will discard any changes. Instead of mapping an element to a database field, you can map it to a literal. For example, you could enter the technical and content contact information directly as literals in the mapping editor dialog (assuming they’re the same for all records). However, we strongly discourage you from doing this. For one thing, this is just bad style, since it will scatter the information published over different places: Some will be retrieved from the database, some (the literals) will be loaded from the configuration files. Changing the metadata of your service later would then require access to the database and access to the BioCASe configuration – that means the configuration tool password. Be nice to the person who will do these changes and put everything neatly in a database table! The second reason is even more important: Using literals for an element will result in this element to become not searchable, which means it cannot be used in a filter of a Search or a Scan request. Even though this behaviour is discoverable with a Capabilities request (by evaluating the ''searchable'' flag), a potential user of your web service might just not think of that, so it’s a good idea to avoid that in first place. Please use literals sparsely! In the mapping editor window, you can select several source columns and several literals, even a combination of both. The Provider Software will simply concatenate the values retrieved from the database and the literals specified (in the order they’re given). But be careful: any use of concatenation or literals will make the element not searchable. A better solution is to do such manipulations earlier, in a database view, and direct the mapping editor to a single resulting database field. Use the mapping editor window to add mappings for all eight mandatory fields: Click on the <strong>+</strong> symbol next to the element, choose table/column name and the correct data type, then press <small>Save</small>. When you’re done, the page should look similar to the following image. Press <small>Save</small> in the main window to write your changes to the configuration files. [[File:abcdMappingEditorMandatoryElements.png|border]] == Initial Testing of the Data Source == Once you’ve mapped all mandatory elements, you can do the first test of your BioCASe web service. Even though you haven’t mapped any real information yet, it is a good idea to check the service and solve any problems hitherto before continuing with the mapping. Click on <small>Test mapping!</small> to open the Query Form that can be used for this purpose: [[File:abcdQueryFormEmpty.png|border]] With the Query Form you can send a BioCASe request to your web service; it will wait for the reply of the web service and display the response document. On top of the page you can see the URL of your web service. In this case it is ''<nowiki>http://localhost/biocase/pywrapper.cgi?dsa=flora</nowiki>'', ''flora'' being the name of the datasource. Make sure this is correct for your installation! Into the text box below you can enter the request XML text. Because it would be cumbersome to enter the whole BioCASe request, you can use the links below the box to load templates. For now, click on <small>ABCD2 Search</small> to fill the box with a BioCASe Search request for documents in the ABCD2 format (the one you’ve just mapped): <syntaxhighlight lang="XML"> <?xml version='1.0' encoding='UTF-8'?> <request xmlns='http://www.biocase.org/schemas/protocol/1.3'> <header><type>search</type></header> <search> <requestFormat>http://www.tdwg.org/schemas/abcd/2.06</requestFormat> <responseFormat start='0' limit='10'>http://www.tdwg.org/schemas/abcd/2.06</responseFormat> <filter> <like path='/DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString'>A*</like> </filter> <count>false</count> </search> </request> </syntaxhighlight> As you can see, there is a filter set on the ''ScientificName'' element. Since we haven’t mapped this element yet, this won’t work. So let’s remove the filter (all lines between <filter> and </filter>, including the ''filter'' tags), leaving the text box like this: [[File:abcdQueryFormFilled.png|border]] Pressing <small>Submit</small> will send the request to the web service and – hopefully – display the response document with the correct ABCD records. For the mapping we’ve just created, it should look similar to the following (use the "–" symbol next to the trees TechnicalContact, ContentContact and Metadata to collapse them): [[File:abcdResponseMandatoryElements.png|border]] If you get an error, please check out the [[Debugging]] tutorial on how to debug a web service. You should continue the mapping process only after your web service returns results for the basal mapping we’ve just created. == Adding additional ABCD Elements == Once you’ve successfully managed to map the mandatory ABCD elements ("successfully" means the web service returns ABCD documents), you can go on with the mapping process. This is usually an iterative process: You add a handful of elements (for example, country, country code, locality text and coordinates for the gathering site), then you use the Query Form to see if the information ends up where it is supposed to do. This is the reason the Query Form opens up in a separate tab (or window, of you configured your web browser so): In one tab, you use the mapping editor to add new elements; after pressing <small>Save</small> you go to the Query Form tab and see what the response documents look like. Redo this until you’ve mapped all information you want to publish. In order to map new ABCD concepts, tick the check box labelled <small>Show all concepts</small> and press <small>Refresh</small>. The mapping editor will then display the whole ABCD tree, which looks admittedly a bit intimidating. In order to find the concept you’re looking for, use the search function of your browser. If you want to map the country name of the gathering site, for example, hit <small>Ctrl + F</small> (for Firefox, Microsoft IE and Opera) and type in the name of the ABCD element, namely ''country''. The browser will jump to the ''Country'' tree of the ''Gathering'' element, where you’ll find the elements for country name (''Name''), ISO country code (''ISO3166Code'') and the country name in the country’s language (''NameDerived''): [[File:abcdMappingEditorSearch.png|500px]] Continue the sequence mapping new elements/testing until you’ve mapped all information you want to publish. You can have a look at the list of recommend and commonly used ABCD elements on the [[CommonABCD2Concepts|Common ABCD2 Concepts]] and [[ABCD2Concepts]] pages. The full ABCD documentation can be found in the [http://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. The following page will show you a sample response ABCD document for a web service with 77 concepts mapped. Of course your documents will look different, because you will probably use different ABCD elements, but still it’s good to get an idea: [[SampleABCDDocument]]. 373ccf266ecb5fb0f99689ef83cac1ee74cafbe2 0 80 1069 917 2018-10-01T09:04:04Z JoergHoletschek 3 /* Discovery of Archives */ wikitext text/x-wiki '''This page describes the archiving features in version 3.4 or later.<br>If you're using version 3.0 or 3.1, please read the [[Archiving_v3.0/3.1|version 3.0/3.1 page]], for versions 3.2 and 3.3, go [http://wiki.bgbm.org/bps/index.php?title=Archiving&oldid=713 here].''' With the archiving feature, you can create archive files that store all data published by your BioCASe web service. That might be handy if your database stores a huge number of records (several hundred thousand or even millions) and the traditional harvesting approach of paging through the records published by the web service takes unacceptably long. Once stored in archives, these files can be downloaded from your web server or sent to the harvester via email and ingested much faster. The OpenUp! natural history aggregator, for example, can use this method for fast harvesting of huge datasets. There are two types of archives: <strong>XML Archives</strong> simply store the records as compressed XML documents, each holding a customizable number of records. If your web service publishes 500,000 records by using the ABCD data schema, for example, an ABCD Archive created for that service could hold 500 ABCD documents, each storing 1,000 records. For each schema supported by a BioCASe web service, separate XML archives can be created. You can download a sample ABCD Archive here: [[File:AlgenEngelsSmall_ABCD_2.06.zip]]. The BioCASe Provider Software can create XML archives for each schema supported by a web service. For the ABCD 2.06 and HISPID 5 schemas, the XML Archiving is dataset-aware, that means each dataset published by the web service will end up in a separate archive file. All other schemas (including ABCD 1.2) will result in one single file for a given web service. <strong>DarwinCore Archives</strong> (DwCAs), in contrast, consist of one or several text files storing the core records and related information, zipped up together with a descriptor file and a metadata document. A detailed specification can be found on the [http://rs.tdwg.org/dwc/terms/guides/text/index.htm TDWG site], a sample archive here: [[File:Desmidiaceae_Engels.zip]]. Information published by web services using the ABCD 2.06 or HISPID 5 schema (dubbed as "ABCD/HISPID dumps") can be stored as DarwinCore Archives. Unsurprisingly, DwCAs make use of the [http://rs.tdwg.org/dwc/terms/index.htm DarwinCore data standard], which is flat and less complex than ABCD 2.06 or HISPID 5. Consequently, a DarwinCore archive will store less information than the corresponding XML archive for a given dataset. ==Managing Archives== The section <small>Existing Archives</small> allows you download, remove and convert archives or view the log of the archiving process: [[File:xmlArchivesNew.png|border|775px]] The archives are grouped by schema, with DarwinCore archives making up an additional group. You can use the minus or plus symbol next to the group header to expand or collapse all files of that group; for web services publishing dozens of datasets, this might be handy to reduce scrolling on the page. The <small>Log </small> link in the group header will open the log file for the respective archiving process in a separate tab. For each archive file, the number of records stored in this file, the date of creation and the file size are listed. XML archives may have a number printed in the <small>Dropped</small> column, meaning some records have been dropped due to missing mandatory data elements. This doesn't mean the archive is invalid, but you should review the mapping or your source data to find the reasons for these lost records. Clicking the archive name will download the archive file, so it can be saved to disk or opened with an archive viewing tool. Pressing the trash symbol at the very right will delete the archive; if you use the trash symbol in the group header, all archives of this schema/type will be removed. For ABCD 2.06 and HISPID 5 XML archives, you will see an additional <small>DwC</small> link next to the trash symbol. This allows you to selectively convert a single dataset from XML to DarwinCore format. In contrast to using the <small>Create DwC Archive</small> button described below, this will spare the XML archiving of the whole web service and just create a DarwinCore archive for the selected dataset, which will be significantly faster. If you use the link in the group header, all archives of the schema will be converted (again without triggering XML archiving first). ==XML Archives== If you’ve finished mapping a schema (see [[ABCD2Mapping]] to learn how to map ABCD, for example) and tested the resulting web service, you can create XML Archives for that web service. To go to the archiving page, simply use the <small>Archive</small> link on the overview page of the datasource setup. If you haven’t mapped a schema, you won’t be able to create any archives. Before archiving, you should always make sure the web service returns correct results by using the QueryForms. The section <small>Create/Update Archives</small> allows you to create new XML archive(s) for each schema mapped. Simply select the schema and press <small>Create XML Archive</small>. As mentioned above, archiving ABCD 2.06 or HISPID 5 is dataset-aware, resulting in one XML archive per dataset published. For all other schemas, a single archive file will be created. The paging size specifies how many records will be bundled into one XML document inside the archive; the default 1,000 should be suitable for most cases. Be aware that a BioCASe web service can restrict the number of records per request - if that limit is below the paging size set for archiving, the paging size will be adjusted automatically after the first packet of records was received from the web service. During the archiving process, the <small>Log</small> will display messages indicating the progress: [[File:xmlArchivesProgressNew.png|border]] You can always cancel the archiving process by pressing the <small>Cancel</small> link in the Status/Messages box. However, it will take some time for the cancelling to show effect, namely until the current paging step has been completed. Depending upon the paging size set and the speed of your server, this can be up to one minute. As the archiving progresses, the first lines of the log will drop out of view. Pressing <small>Show full log</small> will open the full log in a separate browser tab. You can navigate away from the archiving page during the process (or simply close it) and return later. Once finished, the dump will show up in the <small>Existing Archives</small> list, or, if it’s still being processed, as <small>Processing</small> in the Status/Messages box. If the process failed for some reason (or you decided to cancel it), a new box <small>Logs of aborted/failed runs</small> will be displayed with a link for displaying the log file. This will allow you to find out what caused the abortion. For a given web service, you can only create XML archives for one schema at a time. Even though you can start archiving for several web services of your BioCASe installation in parallel, it is not a good idea. The archiving process will put heavy load on your servers – both the database server for data retrieval and the BioCASe server for creating the XML documents. So it is advisable to create just one archive at a time in order to allow both servers to respond to other requests. ==DarwinCore Archives== For web services that use the ABCD 2.06 or HISPID 5 schema, all information published by the web service can be stored in DarwinCore Archives. The DarwinCore data standard is less complex than ABCD, so in most cases the DarwinCore Archive will not store all information published by the web service. DarwinCore archives store only one dataset; so as with ABCD 2.06 and HISPID 5 XML archives, DwC-archiving a web service might result in several DarwinCore archives. DarwinCore archives are not created directly, but from the respective ABCD/HISPID archives. So DwC-archiving a web service will first create ACBD archives, which will be converted into the respective DwC archives subsequently. ===Preparation=== The conversion of ABCD into DwC is done through the Java-based Kettle library. Therefore, you need to have a Java Runtime Environment (version 1.5 or later) installed. You can check this on the library test page under ''Optional external binaries'': [[File:dwcaJavaTest.png|border|500px]] If you don't have one, you can get it [http://www.java.com/de/download/ here]. Important: If you're running BioCASe on a 64bit machine, make sure to get a 64bit Java, since this will boost performance considerably. Once Java is installed, go to the System Administration (<small>Start > Config Tool > System Administration</small>) to check and adapt the conversion settings to your environment: [[File:dwcaSystemSettings.png|600px]] '''Java binary:''' If you're sure Java is installed, but the test lib page shows ''not installed'', please provide the full path to the Java binary (for example ''/usr/local/jdk1.6.0/bin/java''). If you have several Java versions installed (for example 32bit and 64bit versions), make sure to point to the correct (preferably 64bit) version. '''Max memory usage for Java VM (MB):''' This is the maximum amount of heap memory Java will be allowed to use (in MB). The default of 1024 will be sufficient for small datasets (up to 100,000 records). For medium-sized datasets (up to 1m records), you should set this value 2048. For large datasets with millions of records, a value of 4096 is recommended. If your server has enough free memory, you should be generous with this limit - larger values speed up the transformation process on most machines. However, you should make sure to leave enough memory for other required applications on your server. '''Sort buffer size for transformation (number of rows):''' This is the number of rows kept in memory during sorting steps. The default of 100,000 should be OK for most purposes. If you run into memory problems and cannot increase the memory usage, you can lower this value to 50,000 or even 10,000 (do not enter thousands separators into the box). Even lower values will result in heavy disk usage and poor performance. ===Creating an Archive=== DwC archives can only be created for web services that support either ABCD 2.06 or HISPID 5. For all other schemas, the button <small>Create DwC Archive</small> will be disabled. Creating a DwC archive will always create an ABCD archive first, which will then be transformed into DwC. So any existing ABCD archive will be overwritten with an updated version, and any existing DwC archives will be replaced by new versions. To start the archiving process, make sure ABCD 2.06 is selected as ''destination schema'' (or HISPID 5, if you're using that) and press <small>Create DwC Archive</small>. This will start the XML archiving process and, once that is successfully finished, the DwC archive transformation. If you've already created an XML archive before, you can skip the XML archiving and convert this file directly by clicking the ''DwC'' link next to the archive name listed under ''Existing Archives''. The XML archiving step and the parameter <small>Paging size</small> are described in detail in the [[Archiving#XML_Archives|XML Archiving section]] above. During the DwC transformation, the log will display the progress and look similar to this: [[File:dwcaArchiveProgressNew.png|border]] You can cancel the transformation process by clicking the <small>Cancel</small> link in the Status/Messages box. When you do this, BioCASe will try to terminate the process. Depending on the operating system and machine characteristics, this can take some time. Please be patient and wait for the message ''Process terminated'' to appear in the status box. During the process, the first lines of the log output will drop out of view. You can use the <small>Show full log</small> link to open the full log in a separate browser tab. During the conversion process, you can navigate away from the page or close it. When you return and the conversion is still running, you will see the current log output again. If the conversion failed or was cancelled, the log can be viewed from the ''Logs of aborted/failed runs'' box that appears. If the DwC archiving was successful, the line ''SUCCESS: Conversion finished.'' will be printed at the end of the output. The archive(s) generated will be named after the dataset titles and show up under ''Existing archives'', with links for downloading/removing and for re-opening the log file. ===Trouble Shooting=== If you see the line ''ERROR: The DwC Archive creation failed!'', something went wrong. When trying to find the error, always have a look at the full log file instead of just the few last lines printed in the log output box. Either use the ''Show full log'' link or the ''Show'' link below ''Logs of aborted/failed runs''. Start reading the log from top to bottom and find the first error (a line starting with ''ERROR''). Once you've solved this first problem, don't pay attention to any follow-up errors, since they might be caused by the first problem. Instead, restart the transformation using the ''DwC'' link of the respective XML archive and see if the problem is gone. Below are some typical error messages: <syntaxhighlight> [Errno 13] Permission denied: 'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\FloraExsiccataBavarica_DwCA_1.0.log' </syntaxhighlight> Access is denied for the archiving temp directory. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> ERROR 05-07 17:06:47,374 - Error trying to remove old status file: [Error 5] Access denied: u'C:\\Workspace\\bps2\\archive\\FloraExsiccataBavarica\\duplicate.txt' ERROR 05-07 17:06:52,372 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Same problem as above, status file ''duplicate.txt'' cannot be removed. <syntaxhighlight> INFO 05-07 16:41:14,386 - Starting the transformation engine INFO 05-07 16:41:14,386 - Java heap space used is 6144M, sort size is 100000 ERROR 05-07 16:41:14,388 - Java process could not be started: [Error 2] Das System kann die angegebene Datei nicht finden ERROR 05-07 16:41:14,389 - Check the Test libs page to make sure Java is properly installed and configured. ERROR 05-07 16:41:19,384 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Well - what it reads. <syntaxhighlight> INFO 05-07 16:43:42,892 - Starting the transformation engine INFO 05-07 16:43:42,892 - Java heap space used is 1,024M, sort size is 100000 Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Invalid maximum heap size: -Xmx1,024m INFO 05-07 16:43:42,908 - Transformation ended with return code 1 ERROR 05-07 16:43:47,891 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Please do not use thousands separators or any unit (MB, GB) for the maximum heap space, just a number: ''1024''. <syntaxhighlight> ... INFO 05-07 16:50:45,359 - Transformation - Loading transformation from XML file [file:///C:/Workspace/bps2/lib/biocase/archive/abcd2.ktr] INFO 05-07 16:50:45,610 - DwC Conversion - Dispatching started for transformation [DwC Conversion] INFO 05-07 16:50:45,668 - DwC Conversion - This transformation can be replayed with replay date: 2012/07/05 16:50:45 INFO 05-07 16:50:45,710 - Get file names - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) INFO 05-07 16:50:45,718 - Get parameters - Finished processing (I=0, O=0, R=47, W=329, U=0, E=0) ERROR 05-07 16:50:55,904 - Sort meta - UnexpectedError: ERROR 05-07 16:51:16,093 - Sort meta - java.lang.OutOfMemoryError: GC overhead limit exceeded ... </syntaxhighlight> Not enough memory. Go to the System Administration page and increase the memory limit (minimum is 512) or reduce the number of rows kept in memory during sorting. <syntaxhighlight> ... INFO 05-07 16:57:08,435 - Sort occurrences - Finished processing (I=0, O=0, R=2179, W=2179, U=0, E=0) ERROR 05-07 16:57:08,438 - Abort - Row nr 1 causing abort : [General], [response.1.xml], [file:///C:/Workspace/bps2/archive/FloraExsiccataBavarica/tmp/response.1.xml], [DwC Archive creation], [Unzip file], [null], [2012/07/05 16:57:04.129], [100000], [C:\Workspace\bps2\archive\FloraExsiccataBavarica], [RBG], [Flora exsiccata Bavarica], [1591], [Preserved Specimen], [], [], [], [520], [], [], [], [Germany], [DE], [Puchheim westlich von München; Hausmullschutt auf Moorboden], [], [], [20.08.1915], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [Flora exsiccata Bavarica], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/tmp/Flora_exsiccata_Bavarica occurrence], [], [C:\Workspace\bps2\archive\FloraExsiccataBavarica/duplicate] ERROR 05-07 16:57:08,438 - Abort - Duplicate catalog number found. ... INFO 05-07 16:57:09,762 - Kitchen - Processing ended after 5 seconds. INFO 05-07 16:57:09,813 - Transformation ended with return code 1 ERROR 05-07 16:57:09,815 - ERROR: The DwC Archive creation failed! ERROR 05-07 16:57:09,817 - ERROR: Duplicate catalog number "1591" found in the dataset. Please eliminate it, recreate the XML archive and retry. </syntaxhighlight> The ABCD archive contained a duplicate catalog number "1591", which would make the DwC archive invalid. Eliminate that from the database, recreate the XML archive and restart the transformation. <syntaxhighlight> ... INFO 05-07 16:37:01,677 - Removing old files from download folder ERROR 05-07 16:37:01,679 - [Error 5] Access denied: u'C:\\Workspace\\bps2\\www\\downloads\\EDIT_ATBI\\EDIT_-_ATBI_in_Borjomi_Kharagauli_Georgia.dwca.zip' ERROR 05-07 16:37:01,680 - ERROR: The DwC Archive creation failed! </syntaxhighlight> Access is denied for the download folder. Use the libs test page (section ''Status of writable directories and files'') to make sure Python has full write privileges to this folder. <syntaxhighlight> INFO 05-07 17:55:22,793 - DwC Archive creation - Starting entry [Unzip file] ERROR 05-07 17:57:03,253 - Unzip file - Error trying to process zipped entry [zip:file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip!/response.142.xml] from file [file:///C:/Workspace/bps2/www/downloads/natureinfo/natureinfo_ABCD_2.06.zip] ! ERROR 05-07 17:57:03,253 - Unzip file - java.io.IOException: Not enough space on volume. </syntaxhighlight> During the transformation, the XML archive will be unpacked temporarily. This requires some disk space, which wasn't available in this case. == Automating Archiving == Starting with version 3.3 of the BioCASe Provider Software, archiving can be triggered from outside the configuration tool by using a deep link. This allows for scheduling of the archiving process; for example, an ABCD archive could be created once a month to be harvested by GBIF. Both XML and Darwin Core Archive generation can be triggered; moreover, the status of the archiving engine can be retrieved together with the log of the latest archiving process. ===Usage and Parameters=== Assuming you have a data source called ''pontaurus'' with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, the URL to be used for triggering archiving would be http://localhost/biocase/archiving.cgi?dsa=pontaurus. The following lists the parameters that can be specified. Only ''dsa'' and ''pw'' are mandatory. ;dsa (Mandatory): Specifies the data source you want to administer. ;pw (Mandatory): Config tool password, either for the whole installation or the data source, if a data source specific password has been set. ;action (Optional): This specifies the task to be done. Can be one of the values ''log'', ''cancel'', ''xml'' and ''dwc'', each with the following meaning: :{| class="wikitable" |- ! Value ! Description |- | log | This is the default action. It returns the status of the archiving engine for the given data source and lists the log for the latest archiving process. |- | cancel | Cancels any running archiving process for the given data source. If no process is running, it will do nothing. |- | xml | Starts XML archiving. You can customize the process with the optional parameters ''schema'' and ''filesize''. |- | dwc | Starts XML archiving with subsequent DarwinCore Archive transformation. You can customize the process with the optional parameters ''schema'' and ''filesize''. Remember that Java needs to be installed for DarwinCore Archiving. |} ;schema (Optional): Data schema that will be used for storing the published records in the archive. The existing schemas for a data source are listed in the <small>Schemas</small> section on the overview page of the data source configuration. Sample values would be ''ABCD_2.06'' and ''HISPID_5'' (note that the trailing .xml must not be included in this parameter). The default schema that will be used is ABCD 2.06 (identified by schema namespace ''http://www.tdwg.org/schemas/abcd/2.06''); so if that’s what you want, there is no need to specify this parameter. Also no need to specify this parameter if only one schema is mapped; BioCASe will use this schema mapping file. :HISPID 5 users: If several schemas are mapped for the data source and the default schema ABCD 2.06 is not used, BioCASe will try to find a mapping file for HISPID 5 (identified by namespace ''http://www.chah.org.au/schemas/hispid/5''); so if you have HISPID 5 mapped, but not ABCD 2.06, you can also omit this parameter and let BioCASe find the mapping file for you. ;filesize (Optional): Number of records to be stored in a single file of the archive; default is 1,000. If the number of records per request has been limited for the given data source to a value smaller than this, the file size will be adjusted automatically. This parameter corresponds to the paging size that can be set on the archiving page. ===Return values of the script=== ;For action ''cancel'': The script will return a message once the process has been cancelled successfully: ''Process cancelled.'' ;For action ''log'': The log returned will look similar to this: <pre> Idle. Below you'll find the log of the latest archiving process. ************************************************************************************************************************ INFO 02-10 11:51:49,527 - Starting the transformation engine INFO 02-10 11:51:49,527 - Java heap space used is 6144M, sort size is 100000 INFO 02-10 11:51:55,299 - Using "C:\Workspace\bps2\archive\BoBO\tmp\vfs_cache" as temporary files store. INFO 02-10 11:51:55,439 - Kitchen - Start of run. INFO 02-10 11:51:55,533 - DwC Archive creation - Start of job execution (snipped) Below you'll find the log of the corresponding XML archiving process. ************************************************************************************************************************ INFO 02-10 11:51:47,951 - Archive file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip INFO 02-10 11:51:47,951 - Query file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.query.xml INFO 02-10 11:51:47,951 - Config file: C:\Workspace\bps2\archive\BoBO\BoBO_ABCD_2.06.zip.config.xml INFO 02-10 11:51:47,951 - Datasource: BoBO INFO 02-10 11:51:47,951 - Download folder: C:\Workspace\bps2\www\downloads\BoBO INFO 02-10 11:51:47,951 - Wrapper URL: http://localhost/biocase/pywrapper.cgi?dsa=BoBO INFO 02-10 11:51:47,951 - Ticket: 80f2963d09b8e9deb6c8f667262df0b9 INFO 02-10 11:51:47,951 - Limit: 1000 INFO 02-10 11:51:47,967 - Requesting records 1 - 1000 INFO 02-10 11:51:48,497 - Hits: 9 INFO 02-10 11:51:48,497 - Count: 9 INFO 02-10 11:51:48,497 - Dropped: 0 INFO 02-10 11:51:48,497 - Status: ok INFO 02-10 11:51:48,497 - Message: response.1.xml stored in archive BoBO_ABCD_2.06.zip INFO 02-10 11:51:49,511 - SUCCESS: Archiving finished. </pre> ;For actions ''xml'' and ''dwc'': If an archiving process has been started successfully, it will not return any result and time out eventually. For these actions, use separate ''log'' requests to find out about the progress of the process. ;Possible error messages: The script will return an error message if the requested action cannot be performed. Below you’ll find the possible errors returned: :*''Invalid Datasource given!'' :*''The specified datasource doesn't have any schemas mapped!'' :*''The specified schema mapping doesn't exist for this datasource!'' :*''No password provided!'' :*''Authentication failed!'' :*''Unrecognized value for parameter action. Only 'log', 'xml', 'dwc' and 'cancel' are allowed.'' :*''DarwinCore archives can only be created for the ABCD 2.06 and HISPID 5 schemas, so make sure the datasource supports one of these. If you don't specify a schema in the request, BioCASe will try to use ABCD 2.06, otherwise HIDPID 5.'' ===Examples=== Assuming you have a data source called pontaurus with the access point http://localhost/biocase/pywrapper.cgi?dsa=pontaurus, valid requests would be ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo: for returning the status of the archiving engine for data source pontaurus (Idle/Running) and the log for the latest archiving process. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml: to start XML archiving for any existing ABCD 2.06 mapping with default file size (1000). ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=xml&filesize=500&schema=HISPID_5: to start XML archiving for HIDPID 5 with filesize=500. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=dwc: to start XML archiving for any existing ABCD 2.06 mapping; after that has finished, this ABCD archive will be converted into DwC archives. ;http://localhost/biocase/archiving.cgi?dsa=pontaurus&pw=foo&action=cancel: to cancel any running XML or DwC archiving process. == Discovery of Archives == Archives available for a given data source can be discovered through the so-called dataset inventory. To get this, just append ''&inventory=1'' to the access URL of the data source, for example http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar&inventory=1. The dataset inventory response looks roughly like this (diagnostic messages have been removed for brevity): <syntaxhighlight lang="XML"> <dsi:inventory xmlns:dsi="http://www.biocase.org/schemas/dsi/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.biocase.org/schemas/dsi/1.0 http://www.bgbm.org/biodivinf/schema/dsi_1_0.xsd"> <!--XML generated by BioCASE PyWrapper software version 3.6.4. Made in Berlin.--> <dsi:status>OK</dsi:status> <dsi:created>2017-11-01T10:58:38.109000</dsi:created> <dsi:service_url>http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=Herbar</dsi:service_url> <dsi:datasets> <dsi:dataset> <dsi:title>Herbarium Berolinense</dsi:title> <dsi:id>Herbarium Berolinense</dsi:id> <dsi:archives> <dsi:archive filesize="20187103" modified="2017-04-20T12:59:33.810767" namespace="http://www.tdwg.org/schemas/abcd/2.06" rcount="206223">http://ww3.bgbm.org/biocase/downloads/Herbar/Herbarium%20Berolinense.ABCD_2.06.zip</dsi:archive> <dsi:archive filesize="12373841" modified="2017-04-20T13:16:03.772798" rcount="206223" rowType="http://rs.tdwg.org/dwc/terms/Occurrence">http://ww3.bgbm.org/biocase/downloads/Herbar/Herbarium%20Berolinense.DwCA.zip</dsi:archive> </dsi:archives> </dsi:dataset> </dsi:datasets> <dsi:diagnostics> ... </dsi:diagnostics> </dsi:inventory> </syntaxhighlight> The dataset inventory will list all datasets published by a given data source, each with its title and ID. If archives exist for this dataset, they will be listed in ''archives'', each ''archive'' element storing the URL for the respective file. For XML archives, the attribute ''namespace'' will store the namespace of the schema, for DarwinCore archives there will an attribute ''rowType'' with the value <nowiki>http://rs.tdwg.org/dwc/terms/Occurrence</nowiki>. The attribute ''rcount'' specifies the number of records stored in the archive file, ''filesize'' and ''modified'' are hopefully self-explanatory. You can also view the dataset inventory using the manual QueryForm: Scroll down to the section ''Configuration Info'' and click on the link ''Inventory''. After a second, it will appear in the text box below. de96d2cf2669b5467d3783b0b9cad77788839d3f 0 1 1070 395 2018-10-01T09:18:04Z JoergHoletschek 3 /* Table of Contents */ wikitext text/x-wiki This is the Documentation Wiki of the BioCASe Provider Software. Here you will find tutorials on how to install, configure and debug the BioCASe Provider Software. More detailed information on BioCASe can be found in the [[BeginnersGuide|Beginner's Guide]] and on the [http://www.biocase.org/whats_biocase BioCASe homepage]. [[Image:BioCASeLogoLarge.jpg|link=http://www.biocase.org|150px|right|BioCASe]]Described in one paragraph, the BioCASe Provider Software is an XML data binding middleware for publishing data from a relational database to an information network. It is agnostic of the data model used for data publication and can be used in conjunction with any conceptual schema. However, its main field of application is the publication of occurrence data from specimen or observational databases to primary biodiversity information networks such as the BioCASe network (http://www.biocase.org) and the Global Biodiversity Information Facility (http://www.gbif.org). After installing BioCASe and configuring it for a given database, the published information will be accessible as a BioCASe web service, which means it can be retrieved with BioCASe protocol requests. == BioCASe Support == [[Image:mail.png|link=Special:Contact|150px|right|BioCASe Support]] Please get in contact with the BioCASe Support if you * have any questions concerning BioCASe in general or the installation/configuration of the Provider Software, * want to file a bug report, * have any comments, or * want to make a suggestion for a useful new feature in upcoming versions. You can either * use the ''Discussion'' tab of each page to leave us a comment, * use the form on the [[Special:Contact|Contact page]] to send us a message, * or directly mail us to the address given on the [[Special:Contact|Contact page]]. == Table of Contents == ;[[BeginnersGuide|Beginner's Guide]]: What the Provider Software is all about. ;[[Preparation]]: Things to be considered when publishing a database with BioCASe and potential preparatory steps needed. [[Image:support_stamp.png|link=Special:Contact|200px|right|BioCASe Support]] ;[[Installation]]: Tutorial on how to install the BioCASe Provider Software. ;[[DatasourceSetup|Datasource Setup]]: How to set up a new BioCASe web service (aka data source). ;[[ABCD2Mapping|ABCD2 Mapping]]: Tutorial on mapping ABCD2. ;[[CommonABCD2Concepts|Common ABCD2 Concepts]], [[ABCD2Concepts|ABCD2 Concepts]]: List of mandatory, recommended and commonly used ABCD2 elements. ;[[SampleABCDDocument|Sample ABCD2 Document]]: Example ABCD2 document of a comprehensively mapped data provider. ;[[Debugging|Debugging Tutorial]]: Instructions on how to debug a BioCASe web service. ;[[Archiving]]: Learn how to create ABCD Archives or DarwinCore Archives of your published data. ;[[Glossary|Glossary]]: List of terms and abbreviations used in this documentation. ;[[FAQ|FAQ]]: List of frequently asked questions and hopefully helpful answers. ;[[VersionHistory|Version History]]: Change list of current and past versions of the BioCASe Provider Software. b3ce55794fc8dad1ff5e25fc948c799782f2085a 8 3 1071 396 2018-10-01T09:21:47Z JoergHoletschek 3 wikitext text/x-wiki * Table of Contents ** mainpage|mainpage-description ** BeginnersGuide|Beginner's Guide ** Preparation|Preparation ** Installation|Installation ** DatasourceSetup|Datasource Setup ** ABCD2Mapping|ABCD Mapping ** Debugging|Debugging ** Archiving|Archiving ** Glossary|Glossary ** FAQ|FAQ * ABCD 2.06 ** CommonABCD2Concepts|Common Elements ** ABCD2Concepts|Incomplete ABCD2 Doc ** SampleABCDDocument|Sample Document * Feedback ** Special:Contact|Contact Us b7380dd285b97db9db3dcf974c727042e9995f26 1072 1071 2018-10-01T09:22:14Z JoergHoletschek 3 wikitext text/x-wiki * Table of Contents ** mainpage|mainpage-description ** BeginnersGuide|Beginner's Guide ** Preparation|Preparation ** Installation|Installation ** DatasourceSetup|Datasource Setup ** ABCD2Mapping|ABCD Mapping ** Debugging|Debugging ** Archiving|Archiving ** Glossary|Glossary ** FAQ|FAQ * ABCD 2.06 ** CommonABCD2Concepts|Common Elements ** ABCD2Concepts|Incomplete Documentation ** SampleABCDDocument|Sample Document * Feedback ** Special:Contact|Contact Us 122e9fc0e4dc2fc7b5443fd0fbf1320e1d9bf7b8 0 115 1073 1063 2018-10-01T09:25:20Z JoergHoletschek 3 /* General unit data */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens/observations. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact* |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact* |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData* |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation* |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} '''Note:''' If you publish your data to GBIF, you are required to specify a license in the ''License'' node with the ''language'' attribute and the elements ''Text'' and ''URI''. The latter should contain a link to a well-defined license, for example https://creativecommons.org/publicdomain/zero/1.0/. More details on GBIF licensing can be found [http://www.gbif.org/terms/licences here]. ==Specimen Data== In ACBD terminology, a single specimen/observation is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |LivingSpecimen |An indication of what the unit record describes; needs to comply with a [http://tdwg.github.io/dwc/terms/index.htm#basisOfRecord controlled vocabulary]. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="9" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |ScientificName |FullScientificNameString* |Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |Complete name of the taxon including Authors (and years for animals). |- |rowspan="5" | ScientificName/NameAtomised/Botanical |GenusOrMonomial |Aaronsohnia |rowspan="5" | ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |- |FirstEpithet |factorovskyi |- |InfraspecificEpithet |factorovskyi |- |Rank |var. |- |AuthorTeam |Warb. & Eig. |- | - |NameComments | |Free text comments related to the name identified for the unit. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===Multimedia=== For providing links to associated images, videos and sounds, the sub-tree ''MultimediaObjects'' can be used. Each ABCD unit can have multiple multimedia objects, each represented by its own ''MultimediaObject'' with the following elements: {| class="wikitable" !ABCD element !Example !Remark |- |FileURI* |http://ww2.bgbm.org/herbarium/images/B/-W/14/61/B_-W_14616%20-00%200.jpg |The Universal Resource Identifier (URI) of the underlying multimedia file. |- |ProductURI |http://ww2.bgbm.org/herbarium/view_biocase.cfm?SpecimenPK=726 |The Universal Resource Identifier of an electronic source of a multimedia object including HTML, Javascript or similar means delivering rendered output. |- |Comment |Image of herbarium specimen E00732880 by Specimen Digitisation Pipeline. |Short description of the multimedia object. |- |Format |image/jpeg |A code or descriptive term for the file or object format. |- |FileSize |2365 |The size in Kb of the file referred to by the URI or filename for the multimedia object. |- |Creator |Paris, Museum National d'Histoire Naturelle, OD |The identifier (e.g. name or other id) of personnel who has taken the image. |- |CreatedDate |2012-07-07 |The date the multimedia object was created. |- |ID |22658 |Identifier for the multimedia object relating to or forming the unit. This may be a filename. |- |IPR | |Intellectual Property Rights relating specifically to the multimedia object. This subtree has the same elements as ''Metadata/IPRStatements''. Fore more details, see the respective [[ABCD2Concepts#General_metadata | remarks]]. |- |} 5a87e51baddc70225e2042e905deba323d60fcbc 1079 1073 2019-02-18T11:00:03Z JoergHoletschek 3 /* General unit data */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens/observations. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact* |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact* |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData* |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation* |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} '''Note:''' If you publish your data to GBIF, you are required to specify a license in the ''License'' node with the ''language'' attribute and the elements ''Text'' and ''URI''. The latter should contain a link to a well-defined license, for example https://creativecommons.org/publicdomain/zero/1.0/. More details on GBIF licensing can be found [http://www.gbif.org/terms/licences here]. ==Specimen Data== In ACBD terminology, a single specimen/observation is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |LivingSpecimen |An indication of what the unit record describes; needs to comply with a [https://dwc.tdwg.org/terms/#basisOfRecord controlled vocabulary]. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="9" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |ScientificName |FullScientificNameString* |Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |Complete name of the taxon including Authors (and years for animals). |- |rowspan="5" | ScientificName/NameAtomised/Botanical |GenusOrMonomial |Aaronsohnia |rowspan="5" | ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |- |FirstEpithet |factorovskyi |- |InfraspecificEpithet |factorovskyi |- |Rank |var. |- |AuthorTeam |Warb. & Eig. |- | - |NameComments | |Free text comments related to the name identified for the unit. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C(urrent) |The current status of the accession in the garden (current or not current). |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===Multimedia=== For providing links to associated images, videos and sounds, the sub-tree ''MultimediaObjects'' can be used. Each ABCD unit can have multiple multimedia objects, each represented by its own ''MultimediaObject'' with the following elements: {| class="wikitable" !ABCD element !Example !Remark |- |FileURI* |http://ww2.bgbm.org/herbarium/images/B/-W/14/61/B_-W_14616%20-00%200.jpg |The Universal Resource Identifier (URI) of the underlying multimedia file. |- |ProductURI |http://ww2.bgbm.org/herbarium/view_biocase.cfm?SpecimenPK=726 |The Universal Resource Identifier of an electronic source of a multimedia object including HTML, Javascript or similar means delivering rendered output. |- |Comment |Image of herbarium specimen E00732880 by Specimen Digitisation Pipeline. |Short description of the multimedia object. |- |Format |image/jpeg |A code or descriptive term for the file or object format. |- |FileSize |2365 |The size in Kb of the file referred to by the URI or filename for the multimedia object. |- |Creator |Paris, Museum National d'Histoire Naturelle, OD |The identifier (e.g. name or other id) of personnel who has taken the image. |- |CreatedDate |2012-07-07 |The date the multimedia object was created. |- |ID |22658 |Identifier for the multimedia object relating to or forming the unit. This may be a filename. |- |IPR | |Intellectual Property Rights relating specifically to the multimedia object. This subtree has the same elements as ''Metadata/IPRStatements''. Fore more details, see the respective [[ABCD2Concepts#General_metadata | remarks]]. |- |} 3b4041ccb8969d4b222e20914fb29d4609d8e33a 1080 1079 2019-04-02T08:21:15Z JoergHoletschek 3 /* Botanical Garden Unit */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens/observations. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact* |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact* |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData* |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation* |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} '''Note:''' If you publish your data to GBIF, you are required to specify a license in the ''License'' node with the ''language'' attribute and the elements ''Text'' and ''URI''. The latter should contain a link to a well-defined license, for example https://creativecommons.org/publicdomain/zero/1.0/. More details on GBIF licensing can be found [http://www.gbif.org/terms/licences here]. ==Specimen Data== In ACBD terminology, a single specimen/observation is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |LivingSpecimen |An indication of what the unit record describes; needs to comply with a [https://dwc.tdwg.org/terms/#basisOfRecord controlled vocabulary]. |- |KindOfUnit |Whole plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="9" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |ScientificName |FullScientificNameString* |Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |Complete name of the taxon including Authors (and years for animals). |- |rowspan="5" | ScientificName/NameAtomised/Botanical |GenusOrMonomial |Aaronsohnia |rowspan="5" | ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |- |FirstEpithet |factorovskyi |- |InfraspecificEpithet |factorovskyi |- |Rank |var. |- |AuthorTeam |Warb. & Eig. |- | - |NameComments | |Free text comments related to the name identified for the unit. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C |The current status of the accession in the garden. Possible values are * C (current accession in the living collection) * D (non-current due to death) * T (non current due to transfer to another record system, normally another garden) * S (stored in a dormant state) * O (other accession status - different from those above) |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===Multimedia=== For providing links to associated images, videos and sounds, the sub-tree ''MultimediaObjects'' can be used. Each ABCD unit can have multiple multimedia objects, each represented by its own ''MultimediaObject'' with the following elements: {| class="wikitable" !ABCD element !Example !Remark |- |FileURI* |http://ww2.bgbm.org/herbarium/images/B/-W/14/61/B_-W_14616%20-00%200.jpg |The Universal Resource Identifier (URI) of the underlying multimedia file. |- |ProductURI |http://ww2.bgbm.org/herbarium/view_biocase.cfm?SpecimenPK=726 |The Universal Resource Identifier of an electronic source of a multimedia object including HTML, Javascript or similar means delivering rendered output. |- |Comment |Image of herbarium specimen E00732880 by Specimen Digitisation Pipeline. |Short description of the multimedia object. |- |Format |image/jpeg |A code or descriptive term for the file or object format. |- |FileSize |2365 |The size in Kb of the file referred to by the URI or filename for the multimedia object. |- |Creator |Paris, Museum National d'Histoire Naturelle, OD |The identifier (e.g. name or other id) of personnel who has taken the image. |- |CreatedDate |2012-07-07 |The date the multimedia object was created. |- |ID |22658 |Identifier for the multimedia object relating to or forming the unit. This may be a filename. |- |IPR | |Intellectual Property Rights relating specifically to the multimedia object. This subtree has the same elements as ''Metadata/IPRStatements''. Fore more details, see the respective [[ABCD2Concepts#General_metadata | remarks]]. |- |} 2f9865e35fd705fefd09ade9debfbd61faa29b9a 1082 1080 2019-04-18T15:08:08Z JoergHoletschek 3 /* General unit data */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens/observations. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact* |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact* |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData* |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation* |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} '''Note:''' If you publish your data to GBIF, you are required to specify a license in the ''License'' node with the ''language'' attribute and the elements ''Text'' and ''URI''. The latter should contain a link to a well-defined license, for example https://creativecommons.org/publicdomain/zero/1.0/. More details on GBIF licensing can be found [http://www.gbif.org/terms/licences here]. ==Specimen Data== In ACBD terminology, a single specimen/observation is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |LivingSpecimen |An indication of what the unit record describes; needs to comply with a [https://dwc.tdwg.org/terms/#basisOfRecord controlled vocabulary]. |- |KindOfUnit |Seed; (whole) plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="9" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |ScientificName |FullScientificNameString* |Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |Complete name of the taxon including Authors (and years for animals). |- |rowspan="5" | ScientificName/NameAtomised/Botanical |GenusOrMonomial |Aaronsohnia |rowspan="5" | ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |- |FirstEpithet |factorovskyi |- |InfraspecificEpithet |factorovskyi |- |Rank |var. |- |AuthorTeam |Warb. & Eig. |- | - |NameComments | |Free text comments related to the name identified for the unit. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C |The current status of the accession in the garden. Possible values are * C (current accession in the living collection) * D (non-current due to death) * T (non current due to transfer to another record system, normally another garden) * S (stored in a dormant state) * O (other accession status - different from those above) |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===Multimedia=== For providing links to associated images, videos and sounds, the sub-tree ''MultimediaObjects'' can be used. Each ABCD unit can have multiple multimedia objects, each represented by its own ''MultimediaObject'' with the following elements: {| class="wikitable" !ABCD element !Example !Remark |- |FileURI* |http://ww2.bgbm.org/herbarium/images/B/-W/14/61/B_-W_14616%20-00%200.jpg |The Universal Resource Identifier (URI) of the underlying multimedia file. |- |ProductURI |http://ww2.bgbm.org/herbarium/view_biocase.cfm?SpecimenPK=726 |The Universal Resource Identifier of an electronic source of a multimedia object including HTML, Javascript or similar means delivering rendered output. |- |Comment |Image of herbarium specimen E00732880 by Specimen Digitisation Pipeline. |Short description of the multimedia object. |- |Format |image/jpeg |A code or descriptive term for the file or object format. |- |FileSize |2365 |The size in Kb of the file referred to by the URI or filename for the multimedia object. |- |Creator |Paris, Museum National d'Histoire Naturelle, OD |The identifier (e.g. name or other id) of personnel who has taken the image. |- |CreatedDate |2012-07-07 |The date the multimedia object was created. |- |ID |22658 |Identifier for the multimedia object relating to or forming the unit. This may be a filename. |- |IPR | |Intellectual Property Rights relating specifically to the multimedia object. This subtree has the same elements as ''Metadata/IPRStatements''. Fore more details, see the respective [[ABCD2Concepts#General_metadata | remarks]]. |- |} 62fde93dc3a5f8f0b8b132c884836be2e0ff5170 1083 1082 2019-04-18T15:56:58Z JoergHoletschek 3 /* Botanical Garden Unit */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens/observations. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact* |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact* |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData* |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation* |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} '''Note:''' If you publish your data to GBIF, you are required to specify a license in the ''License'' node with the ''language'' attribute and the elements ''Text'' and ''URI''. The latter should contain a link to a well-defined license, for example https://creativecommons.org/publicdomain/zero/1.0/. More details on GBIF licensing can be found [http://www.gbif.org/terms/licences here]. ==Specimen Data== In ACBD terminology, a single specimen/observation is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |LivingSpecimen |An indication of what the unit record describes; needs to comply with a [https://dwc.tdwg.org/terms/#basisOfRecord controlled vocabulary]. |- |KindOfUnit |Seed; (whole) plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="9" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |ScientificName |FullScientificNameString* |Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |Complete name of the taxon including Authors (and years for animals). |- |rowspan="5" | ScientificName/NameAtomised/Botanical |GenusOrMonomial |Aaronsohnia |rowspan="5" | ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |- |FirstEpithet |factorovskyi |- |InfraspecificEpithet |factorovskyi |- |Rank |var. |- |AuthorTeam |Warb. & Eig. |- | - |NameComments | |Free text comments related to the name identified for the unit. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C |The current status of the accession in the garden. Possible values are * C (current accession in the living collection) * D (non-current due to death) * T (non current due to transfer to another record system, normally another garden) * S (stored in a dormant state) * O (other accession status - different from those above) |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===SpecimenUnit/Accessions=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionCatalogue |C |The current status of the accession in the garden. Possible values are * C (current accession in the living collection) * D (non-current due to death) * T (non current due to transfer to another record system, normally another garden) * S (stored in a dormant state) * O (other accession status - different from those above) |- |AccessionDate | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |AccessionNumber |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |} ===Multimedia=== For providing links to associated images, videos and sounds, the sub-tree ''MultimediaObjects'' can be used. Each ABCD unit can have multiple multimedia objects, each represented by its own ''MultimediaObject'' with the following elements: {| class="wikitable" !ABCD element !Example !Remark |- |FileURI* |http://ww2.bgbm.org/herbarium/images/B/-W/14/61/B_-W_14616%20-00%200.jpg |The Universal Resource Identifier (URI) of the underlying multimedia file. |- |ProductURI |http://ww2.bgbm.org/herbarium/view_biocase.cfm?SpecimenPK=726 |The Universal Resource Identifier of an electronic source of a multimedia object including HTML, Javascript or similar means delivering rendered output. |- |Comment |Image of herbarium specimen E00732880 by Specimen Digitisation Pipeline. |Short description of the multimedia object. |- |Format |image/jpeg |A code or descriptive term for the file or object format. |- |FileSize |2365 |The size in Kb of the file referred to by the URI or filename for the multimedia object. |- |Creator |Paris, Museum National d'Histoire Naturelle, OD |The identifier (e.g. name or other id) of personnel who has taken the image. |- |CreatedDate |2012-07-07 |The date the multimedia object was created. |- |ID |22658 |Identifier for the multimedia object relating to or forming the unit. This may be a filename. |- |IPR | |Intellectual Property Rights relating specifically to the multimedia object. This subtree has the same elements as ''Metadata/IPRStatements''. Fore more details, see the respective [[ABCD2Concepts#General_metadata | remarks]]. |- |} ec9dfd553098b7477fba9f7a7f82165d970ff67e 1084 1083 2019-04-18T16:02:46Z JoergHoletschek 3 /* SpecimenUnit/Accessions */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens/observations. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact* |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact* |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData* |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation* |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} '''Note:''' If you publish your data to GBIF, you are required to specify a license in the ''License'' node with the ''language'' attribute and the elements ''Text'' and ''URI''. The latter should contain a link to a well-defined license, for example https://creativecommons.org/publicdomain/zero/1.0/. More details on GBIF licensing can be found [http://www.gbif.org/terms/licences here]. ==Specimen Data== In ACBD terminology, a single specimen/observation is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |LivingSpecimen |An indication of what the unit record describes; needs to comply with a [https://dwc.tdwg.org/terms/#basisOfRecord controlled vocabulary]. |- |KindOfUnit |Seed; (whole) plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="9" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |ScientificName |FullScientificNameString* |Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |Complete name of the taxon including Authors (and years for animals). |- |rowspan="5" | ScientificName/NameAtomised/Botanical |GenusOrMonomial |Aaronsohnia |rowspan="5" | ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |- |FirstEpithet |factorovskyi |- |InfraspecificEpithet |factorovskyi |- |Rank |var. |- |AuthorTeam |Warb. & Eig. |- | - |NameComments | |Free text comments related to the name identified for the unit. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C |The current status of the accession in the garden. Possible values are * C (current accession in the living collection) * D (non-current due to death) * T (non current due to transfer to another record system, normally another garden) * S (stored in a dormant state) * O (other accession status - different from those above) |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===SpecimenUnit/Accessions=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionCatalogue |C |Identifier of an accessions catalogue or numbering scheme |- |AccessionDate |2007-07-07 |Date on which the accession was recorded |- |AccessionNumber |001010680 |Institutional number applied to an accession and recorded in the accessions register. |- |} ===Multimedia=== For providing links to associated images, videos and sounds, the sub-tree ''MultimediaObjects'' can be used. Each ABCD unit can have multiple multimedia objects, each represented by its own ''MultimediaObject'' with the following elements: {| class="wikitable" !ABCD element !Example !Remark |- |FileURI* |http://ww2.bgbm.org/herbarium/images/B/-W/14/61/B_-W_14616%20-00%200.jpg |The Universal Resource Identifier (URI) of the underlying multimedia file. |- |ProductURI |http://ww2.bgbm.org/herbarium/view_biocase.cfm?SpecimenPK=726 |The Universal Resource Identifier of an electronic source of a multimedia object including HTML, Javascript or similar means delivering rendered output. |- |Comment |Image of herbarium specimen E00732880 by Specimen Digitisation Pipeline. |Short description of the multimedia object. |- |Format |image/jpeg |A code or descriptive term for the file or object format. |- |FileSize |2365 |The size in Kb of the file referred to by the URI or filename for the multimedia object. |- |Creator |Paris, Museum National d'Histoire Naturelle, OD |The identifier (e.g. name or other id) of personnel who has taken the image. |- |CreatedDate |2012-07-07 |The date the multimedia object was created. |- |ID |22658 |Identifier for the multimedia object relating to or forming the unit. This may be a filename. |- |IPR | |Intellectual Property Rights relating specifically to the multimedia object. This subtree has the same elements as ''Metadata/IPRStatements''. Fore more details, see the respective [[ABCD2Concepts#General_metadata | remarks]]. |- |} fe7ab25c7e3ddcd0536087921820108162c32762 1085 1084 2019-04-18T16:03:00Z JoergHoletschek 3 /* SpecimenUnit/Accessions */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens/observations. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact* |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact* |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData* |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation* |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} '''Note:''' If you publish your data to GBIF, you are required to specify a license in the ''License'' node with the ''language'' attribute and the elements ''Text'' and ''URI''. The latter should contain a link to a well-defined license, for example https://creativecommons.org/publicdomain/zero/1.0/. More details on GBIF licensing can be found [http://www.gbif.org/terms/licences here]. ==Specimen Data== In ACBD terminology, a single specimen/observation is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |LivingSpecimen |An indication of what the unit record describes; needs to comply with a [https://dwc.tdwg.org/terms/#basisOfRecord controlled vocabulary]. |- |KindOfUnit |Seed; (whole) plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="9" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |ScientificName |FullScientificNameString* |Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |Complete name of the taxon including Authors (and years for animals). |- |rowspan="5" | ScientificName/NameAtomised/Botanical |GenusOrMonomial |Aaronsohnia |rowspan="5" | ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |- |FirstEpithet |factorovskyi |- |InfraspecificEpithet |factorovskyi |- |Rank |var. |- |AuthorTeam |Warb. & Eig. |- | - |NameComments | |Free text comments related to the name identified for the unit. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C |The current status of the accession in the garden. Possible values are * C (current accession in the living collection) * D (non-current due to death) * T (non current due to transfer to another record system, normally another garden) * S (stored in a dormant state) * O (other accession status - different from those above) |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===SpecimenUnit/Accessions=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionCatalogue | |Identifier of an accessions catalogue or numbering scheme |- |AccessionDate |2007-07-07 |Date on which the accession was recorded |- |AccessionNumber |001010680 |Institutional number applied to an accession and recorded in the accessions register. |- |} ===Multimedia=== For providing links to associated images, videos and sounds, the sub-tree ''MultimediaObjects'' can be used. Each ABCD unit can have multiple multimedia objects, each represented by its own ''MultimediaObject'' with the following elements: {| class="wikitable" !ABCD element !Example !Remark |- |FileURI* |http://ww2.bgbm.org/herbarium/images/B/-W/14/61/B_-W_14616%20-00%200.jpg |The Universal Resource Identifier (URI) of the underlying multimedia file. |- |ProductURI |http://ww2.bgbm.org/herbarium/view_biocase.cfm?SpecimenPK=726 |The Universal Resource Identifier of an electronic source of a multimedia object including HTML, Javascript or similar means delivering rendered output. |- |Comment |Image of herbarium specimen E00732880 by Specimen Digitisation Pipeline. |Short description of the multimedia object. |- |Format |image/jpeg |A code or descriptive term for the file or object format. |- |FileSize |2365 |The size in Kb of the file referred to by the URI or filename for the multimedia object. |- |Creator |Paris, Museum National d'Histoire Naturelle, OD |The identifier (e.g. name or other id) of personnel who has taken the image. |- |CreatedDate |2012-07-07 |The date the multimedia object was created. |- |ID |22658 |Identifier for the multimedia object relating to or forming the unit. This may be a filename. |- |IPR | |Intellectual Property Rights relating specifically to the multimedia object. This subtree has the same elements as ''Metadata/IPRStatements''. Fore more details, see the respective [[ABCD2Concepts#General_metadata | remarks]]. |- |} 13c3b02adf6eb75cc351e5aa4b9631c8f83a3a6e 1099 1085 2020-04-08T10:27:40Z JoergHoletschek 3 /* Botanical Garden Unit */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens/observations. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact* |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact* |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData* |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation* |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} '''Note:''' If you publish your data to GBIF, you are required to specify a license in the ''License'' node with the ''language'' attribute and the elements ''Text'' and ''URI''. The latter should contain a link to a well-defined license, for example https://creativecommons.org/publicdomain/zero/1.0/. More details on GBIF licensing can be found [http://www.gbif.org/terms/licences here]. ==Specimen Data== In ACBD terminology, a single specimen/observation is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |LivingSpecimen |An indication of what the unit record describes; needs to comply with a [https://dwc.tdwg.org/terms/#basisOfRecord controlled vocabulary]. |- |KindOfUnit |Seed; (whole) plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="9" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |ScientificName |FullScientificNameString* |Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |Complete name of the taxon including Authors (and years for animals). |- |rowspan="5" | ScientificName/NameAtomised/Botanical |GenusOrMonomial |Aaronsohnia |rowspan="5" | ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |- |FirstEpithet |factorovskyi |- |InfraspecificEpithet |factorovskyi |- |Rank |var. |- |AuthorTeam |Warb. & Eig. |- | - |NameComments | |Free text comments related to the name identified for the unit. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C |The current status of the accession in the garden. Possible values are * C (current accession in the living collection) * D (non-current due to death) * T (non current due to transfer to another record system, normally another garden) * S (stored in a dormant state) * O (other accession status - different from those above) |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |DonorCategory |E |A code to indicate the type of the immediate donor from which the accession was obtained. This may not necessarily be the original collector of wild material. One or more of the following characters: E, G, B, R, S, U, H, M, N, I, O, U. For a description of these values, please refer to https://www.bgci.org/files/Databases/itf2.pdf, section E.5 (donor type flag). |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} ===SpecimenUnit/Accessions=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionCatalogue | |Identifier of an accessions catalogue or numbering scheme |- |AccessionDate |2007-07-07 |Date on which the accession was recorded |- |AccessionNumber |001010680 |Institutional number applied to an accession and recorded in the accessions register. |- |} ===Multimedia=== For providing links to associated images, videos and sounds, the sub-tree ''MultimediaObjects'' can be used. Each ABCD unit can have multiple multimedia objects, each represented by its own ''MultimediaObject'' with the following elements: {| class="wikitable" !ABCD element !Example !Remark |- |FileURI* |http://ww2.bgbm.org/herbarium/images/B/-W/14/61/B_-W_14616%20-00%200.jpg |The Universal Resource Identifier (URI) of the underlying multimedia file. |- |ProductURI |http://ww2.bgbm.org/herbarium/view_biocase.cfm?SpecimenPK=726 |The Universal Resource Identifier of an electronic source of a multimedia object including HTML, Javascript or similar means delivering rendered output. |- |Comment |Image of herbarium specimen E00732880 by Specimen Digitisation Pipeline. |Short description of the multimedia object. |- |Format |image/jpeg |A code or descriptive term for the file or object format. |- |FileSize |2365 |The size in Kb of the file referred to by the URI or filename for the multimedia object. |- |Creator |Paris, Museum National d'Histoire Naturelle, OD |The identifier (e.g. name or other id) of personnel who has taken the image. |- |CreatedDate |2012-07-07 |The date the multimedia object was created. |- |ID |22658 |Identifier for the multimedia object relating to or forming the unit. This may be a filename. |- |IPR | |Intellectual Property Rights relating specifically to the multimedia object. This subtree has the same elements as ''Metadata/IPRStatements''. Fore more details, see the respective [[ABCD2Concepts#General_metadata | remarks]]. |- |} 2af04a76eb48f839cceb0737301435371bffa79d 1100 1099 2020-04-08T10:36:08Z JoergHoletschek 3 /* Botanical Garden Unit */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens/observations. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact* |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact* |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData* |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation* |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} '''Note:''' If you publish your data to GBIF, you are required to specify a license in the ''License'' node with the ''language'' attribute and the elements ''Text'' and ''URI''. The latter should contain a link to a well-defined license, for example https://creativecommons.org/publicdomain/zero/1.0/. More details on GBIF licensing can be found [http://www.gbif.org/terms/licences here]. ==Specimen Data== In ACBD terminology, a single specimen/observation is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |LivingSpecimen |An indication of what the unit record describes; needs to comply with a [https://dwc.tdwg.org/terms/#basisOfRecord controlled vocabulary]. |- |KindOfUnit |Seed; (whole) plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="9" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |ScientificName |FullScientificNameString* |Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |Complete name of the taxon including Authors (and years for animals). |- |rowspan="5" | ScientificName/NameAtomised/Botanical |GenusOrMonomial |Aaronsohnia |rowspan="5" | ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |- |FirstEpithet |factorovskyi |- |InfraspecificEpithet |factorovskyi |- |Rank |var. |- |AuthorTeam |Warb. & Eig. |- | - |NameComments | |Free text comments related to the name identified for the unit. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C |The current status of the accession in the garden. Possible values are * C (current accession in the living collection) * D (non-current due to death) * T (non current due to transfer to another record system, normally another garden) * S (stored in a dormant state) * O (other accession status - different from those above) |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |DonorCategory |E |A code to indicate the type of the immediate donor from which the accession was obtained. This may not necessarily be the original collector of wild material. One or more of the following characters: E, G, B, R, S, U, H, M, N, I, O, U. For a description of these values, please refer to https://www.bgci.org/files/Databases/itf2.pdf, section E.5 (donor type flag). Gardens4Science data providers should also see the note below. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} '''[[Note for Gardens4Science data providers:]]''' Whether or not a given record will be visible in the public view of the [http://gardens4science.biocase.org Gardens4Science data portal] (that is, without being logged in) is determined by the value of the DonorCategory field. ===SpecimenUnit/Accessions=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionCatalogue | |Identifier of an accessions catalogue or numbering scheme |- |AccessionDate |2007-07-07 |Date on which the accession was recorded |- |AccessionNumber |001010680 |Institutional number applied to an accession and recorded in the accessions register. |- |} ===Multimedia=== For providing links to associated images, videos and sounds, the sub-tree ''MultimediaObjects'' can be used. Each ABCD unit can have multiple multimedia objects, each represented by its own ''MultimediaObject'' with the following elements: {| class="wikitable" !ABCD element !Example !Remark |- |FileURI* |http://ww2.bgbm.org/herbarium/images/B/-W/14/61/B_-W_14616%20-00%200.jpg |The Universal Resource Identifier (URI) of the underlying multimedia file. |- |ProductURI |http://ww2.bgbm.org/herbarium/view_biocase.cfm?SpecimenPK=726 |The Universal Resource Identifier of an electronic source of a multimedia object including HTML, Javascript or similar means delivering rendered output. |- |Comment |Image of herbarium specimen E00732880 by Specimen Digitisation Pipeline. |Short description of the multimedia object. |- |Format |image/jpeg |A code or descriptive term for the file or object format. |- |FileSize |2365 |The size in Kb of the file referred to by the URI or filename for the multimedia object. |- |Creator |Paris, Museum National d'Histoire Naturelle, OD |The identifier (e.g. name or other id) of personnel who has taken the image. |- |CreatedDate |2012-07-07 |The date the multimedia object was created. |- |ID |22658 |Identifier for the multimedia object relating to or forming the unit. This may be a filename. |- |IPR | |Intellectual Property Rights relating specifically to the multimedia object. This subtree has the same elements as ''Metadata/IPRStatements''. Fore more details, see the respective [[ABCD2Concepts#General_metadata | remarks]]. |- |} 4310e8db2b86aa8b9b5d2e13d7efe7ee6e53f3cc 1101 1100 2020-04-08T10:36:30Z JoergHoletschek 3 /* Botanical Garden Unit */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens/observations. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact* |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact* |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData* |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation* |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} '''Note:''' If you publish your data to GBIF, you are required to specify a license in the ''License'' node with the ''language'' attribute and the elements ''Text'' and ''URI''. The latter should contain a link to a well-defined license, for example https://creativecommons.org/publicdomain/zero/1.0/. More details on GBIF licensing can be found [http://www.gbif.org/terms/licences here]. ==Specimen Data== In ACBD terminology, a single specimen/observation is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |LivingSpecimen |An indication of what the unit record describes; needs to comply with a [https://dwc.tdwg.org/terms/#basisOfRecord controlled vocabulary]. |- |KindOfUnit |Seed; (whole) plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="9" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |ScientificName |FullScientificNameString* |Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |Complete name of the taxon including Authors (and years for animals). |- |rowspan="5" | ScientificName/NameAtomised/Botanical |GenusOrMonomial |Aaronsohnia |rowspan="5" | ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |- |FirstEpithet |factorovskyi |- |InfraspecificEpithet |factorovskyi |- |Rank |var. |- |AuthorTeam |Warb. & Eig. |- | - |NameComments | |Free text comments related to the name identified for the unit. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C |The current status of the accession in the garden. Possible values are * C (current accession in the living collection) * D (non-current due to death) * T (non current due to transfer to another record system, normally another garden) * S (stored in a dormant state) * O (other accession status - different from those above) |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |DonorCategory |E |A code to indicate the type of the immediate donor from which the accession was obtained. This may not necessarily be the original collector of wild material. One or more of the following characters: E, G, B, R, S, U, H, M, N, I, O, U. For a description of these values, please refer to https://www.bgci.org/files/Databases/itf2.pdf, section E.5 (donor type flag). Gardens4Science data providers should also see the note below. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} '''Note for Gardens4Science data providers:''' Whether or not a given record will be visible in the public view of the [http://gardens4science.biocase.org Gardens4Science data portal] (that is, without being logged in) is determined by the value of the DonorCategory field. ===SpecimenUnit/Accessions=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionCatalogue | |Identifier of an accessions catalogue or numbering scheme |- |AccessionDate |2007-07-07 |Date on which the accession was recorded |- |AccessionNumber |001010680 |Institutional number applied to an accession and recorded in the accessions register. |- |} ===Multimedia=== For providing links to associated images, videos and sounds, the sub-tree ''MultimediaObjects'' can be used. Each ABCD unit can have multiple multimedia objects, each represented by its own ''MultimediaObject'' with the following elements: {| class="wikitable" !ABCD element !Example !Remark |- |FileURI* |http://ww2.bgbm.org/herbarium/images/B/-W/14/61/B_-W_14616%20-00%200.jpg |The Universal Resource Identifier (URI) of the underlying multimedia file. |- |ProductURI |http://ww2.bgbm.org/herbarium/view_biocase.cfm?SpecimenPK=726 |The Universal Resource Identifier of an electronic source of a multimedia object including HTML, Javascript or similar means delivering rendered output. |- |Comment |Image of herbarium specimen E00732880 by Specimen Digitisation Pipeline. |Short description of the multimedia object. |- |Format |image/jpeg |A code or descriptive term for the file or object format. |- |FileSize |2365 |The size in Kb of the file referred to by the URI or filename for the multimedia object. |- |Creator |Paris, Museum National d'Histoire Naturelle, OD |The identifier (e.g. name or other id) of personnel who has taken the image. |- |CreatedDate |2012-07-07 |The date the multimedia object was created. |- |ID |22658 |Identifier for the multimedia object relating to or forming the unit. This may be a filename. |- |IPR | |Intellectual Property Rights relating specifically to the multimedia object. This subtree has the same elements as ''Metadata/IPRStatements''. Fore more details, see the respective [[ABCD2Concepts#General_metadata | remarks]]. |- |} a1e0a2200ade478a1efd696cddc64a9acdd043e5 1102 1101 2020-04-08T14:07:16Z JoergHoletschek 3 /* Botanical Garden Unit */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens/observations. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact* |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact* |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData* |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation* |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} '''Note:''' If you publish your data to GBIF, you are required to specify a license in the ''License'' node with the ''language'' attribute and the elements ''Text'' and ''URI''. The latter should contain a link to a well-defined license, for example https://creativecommons.org/publicdomain/zero/1.0/. More details on GBIF licensing can be found [http://www.gbif.org/terms/licences here]. ==Specimen Data== In ACBD terminology, a single specimen/observation is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |LivingSpecimen |An indication of what the unit record describes; needs to comply with a [https://dwc.tdwg.org/terms/#basisOfRecord controlled vocabulary]. |- |KindOfUnit |Seed; (whole) plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="9" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |ScientificName |FullScientificNameString* |Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |Complete name of the taxon including Authors (and years for animals). |- |rowspan="5" | ScientificName/NameAtomised/Botanical |GenusOrMonomial |Aaronsohnia |rowspan="5" | ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |- |FirstEpithet |factorovskyi |- |InfraspecificEpithet |factorovskyi |- |Rank |var. |- |AuthorTeam |Warb. & Eig. |- | - |NameComments | |Free text comments related to the name identified for the unit. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C |The current status of the accession in the garden. Possible values are * C (current accession in the living collection) * D (non-current due to death) * T (non current due to transfer to another record system, normally another garden) * S (stored in a dormant state) * O (other accession status - different from those above) |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |DonorCategory |E |A code to indicate the type of the immediate donor from which the accession was obtained. This may not necessarily be the original collector of wild material. One or more of the following characters: E, G, B, R, S, U, H, M, N, I, O, U. For a description of these values, please refer to https://www.bgci.org/files/Databases/itf2.pdf, section E.5 (donor type flag). Gardens4Science data providers should also see the note below. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} '''Note for Gardens4Science data providers:''' Whether or not a given record will be visible in the public view of the [http://gardens4science.biocase.org Gardens4Science data portal] (that is, without being logged in) is determined by the value of the DonorCategory field: Records "O = other" will be visible only for logged-in curators; if the field is filled with "I = individual" or empty, the record will also show up for unregistered users. ===SpecimenUnit/Accessions=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionCatalogue | |Identifier of an accessions catalogue or numbering scheme |- |AccessionDate |2007-07-07 |Date on which the accession was recorded |- |AccessionNumber |001010680 |Institutional number applied to an accession and recorded in the accessions register. |- |} ===Multimedia=== For providing links to associated images, videos and sounds, the sub-tree ''MultimediaObjects'' can be used. Each ABCD unit can have multiple multimedia objects, each represented by its own ''MultimediaObject'' with the following elements: {| class="wikitable" !ABCD element !Example !Remark |- |FileURI* |http://ww2.bgbm.org/herbarium/images/B/-W/14/61/B_-W_14616%20-00%200.jpg |The Universal Resource Identifier (URI) of the underlying multimedia file. |- |ProductURI |http://ww2.bgbm.org/herbarium/view_biocase.cfm?SpecimenPK=726 |The Universal Resource Identifier of an electronic source of a multimedia object including HTML, Javascript or similar means delivering rendered output. |- |Comment |Image of herbarium specimen E00732880 by Specimen Digitisation Pipeline. |Short description of the multimedia object. |- |Format |image/jpeg |A code or descriptive term for the file or object format. |- |FileSize |2365 |The size in Kb of the file referred to by the URI or filename for the multimedia object. |- |Creator |Paris, Museum National d'Histoire Naturelle, OD |The identifier (e.g. name or other id) of personnel who has taken the image. |- |CreatedDate |2012-07-07 |The date the multimedia object was created. |- |ID |22658 |Identifier for the multimedia object relating to or forming the unit. This may be a filename. |- |IPR | |Intellectual Property Rights relating specifically to the multimedia object. This subtree has the same elements as ''Metadata/IPRStatements''. Fore more details, see the respective [[ABCD2Concepts#General_metadata | remarks]]. |- |} 752967e403cc66fb5ec14dbd7e4c9f954f107acf 1107 1102 2021-02-01T10:34:38Z JoergHoletschek 3 /* Botanical Garden Unit */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens/observations. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact* |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact* |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData* |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation* |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} '''Note:''' If you publish your data to GBIF, you are required to specify a license in the ''License'' node with the ''language'' attribute and the elements ''Text'' and ''URI''. The latter should contain a link to a well-defined license, for example https://creativecommons.org/publicdomain/zero/1.0/. More details on GBIF licensing can be found [http://www.gbif.org/terms/licences here]. ==Specimen Data== In ACBD terminology, a single specimen/observation is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |LivingSpecimen |An indication of what the unit record describes; needs to comply with a [https://dwc.tdwg.org/terms/#basisOfRecord controlled vocabulary]. |- |KindOfUnit |Seed; (whole) plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="9" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |ScientificName |FullScientificNameString* |Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |Complete name of the taxon including Authors (and years for animals). |- |rowspan="5" | ScientificName/NameAtomised/Botanical |GenusOrMonomial |Aaronsohnia |rowspan="5" | ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |- |FirstEpithet |factorovskyi |- |InfraspecificEpithet |factorovskyi |- |Rank |var. |- |AuthorTeam |Warb. & Eig. |- | - |NameComments | |Free text comments related to the name identified for the unit. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C |The current status of the accession in the garden. Possible values are * C (current accession in the living collection) * D (non-current due to death) * T (non current due to transfer to another record system, normally another garden) * S (stored in a dormant state) * O (other accession status - different from those above) |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |DonorCategory |E |A code to indicate the type of the immediate donor from which the accession was obtained. This may not necessarily be the original collector of wild material. One or more of the following characters: E, G, B, R, S, U, H, M, N, I, O, U. For a description of these values, please refer to https://www.bgci.org/files/Databases/itf2.pdf, section E.5 (donor type flag). Gardens4Science data providers should also see the note below. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |HybridFlag | * no hybrid: <empty> * named hybrid: * * graft chimaera: + * any other type of hybrid: "hybrid" |A flag indicating that the accession is grown outdoors in the botanical garden. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} '''Note for Gardens4Science data providers:''' Whether or not a given record will be visible in the public view of the [http://gardens4science.biocase.org Gardens4Science data portal] (that is, without being logged in) is determined by the value of the DonorCategory field: Records "O = other" will be visible only for logged-in curators; if the field is filled with "I = individual" or empty, the record will also show up for unregistered users. ===SpecimenUnit/Accessions=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionCatalogue | |Identifier of an accessions catalogue or numbering scheme |- |AccessionDate |2007-07-07 |Date on which the accession was recorded |- |AccessionNumber |001010680 |Institutional number applied to an accession and recorded in the accessions register. |- |} ===Multimedia=== For providing links to associated images, videos and sounds, the sub-tree ''MultimediaObjects'' can be used. Each ABCD unit can have multiple multimedia objects, each represented by its own ''MultimediaObject'' with the following elements: {| class="wikitable" !ABCD element !Example !Remark |- |FileURI* |http://ww2.bgbm.org/herbarium/images/B/-W/14/61/B_-W_14616%20-00%200.jpg |The Universal Resource Identifier (URI) of the underlying multimedia file. |- |ProductURI |http://ww2.bgbm.org/herbarium/view_biocase.cfm?SpecimenPK=726 |The Universal Resource Identifier of an electronic source of a multimedia object including HTML, Javascript or similar means delivering rendered output. |- |Comment |Image of herbarium specimen E00732880 by Specimen Digitisation Pipeline. |Short description of the multimedia object. |- |Format |image/jpeg |A code or descriptive term for the file or object format. |- |FileSize |2365 |The size in Kb of the file referred to by the URI or filename for the multimedia object. |- |Creator |Paris, Museum National d'Histoire Naturelle, OD |The identifier (e.g. name or other id) of personnel who has taken the image. |- |CreatedDate |2012-07-07 |The date the multimedia object was created. |- |ID |22658 |Identifier for the multimedia object relating to or forming the unit. This may be a filename. |- |IPR | |Intellectual Property Rights relating specifically to the multimedia object. This subtree has the same elements as ''Metadata/IPRStatements''. Fore more details, see the respective [[ABCD2Concepts#General_metadata | remarks]]. |- |} b069b3a875ab98f490e2f22d13108fd369e71cc5 1108 1107 2021-02-01T10:36:15Z JoergHoletschek 3 /* Botanical Garden Unit */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens/observations. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact* |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact* |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData* |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation* |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} '''Note:''' If you publish your data to GBIF, you are required to specify a license in the ''License'' node with the ''language'' attribute and the elements ''Text'' and ''URI''. The latter should contain a link to a well-defined license, for example https://creativecommons.org/publicdomain/zero/1.0/. More details on GBIF licensing can be found [http://www.gbif.org/terms/licences here]. ==Specimen Data== In ACBD terminology, a single specimen/observation is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |LivingSpecimen |An indication of what the unit record describes; needs to comply with a [https://dwc.tdwg.org/terms/#basisOfRecord controlled vocabulary]. |- |KindOfUnit |Seed; (whole) plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="9" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |ScientificName |FullScientificNameString* |Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |Complete name of the taxon including Authors (and years for animals). |- |rowspan="5" | ScientificName/NameAtomised/Botanical |GenusOrMonomial |Aaronsohnia |rowspan="5" | ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |- |FirstEpithet |factorovskyi |- |InfraspecificEpithet |factorovskyi |- |Rank |var. |- |AuthorTeam |Warb. & Eig. |- | - |NameComments | |Free text comments related to the name identified for the unit. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C |The current status of the accession in the garden. Possible values are * C (current accession in the living collection) * D (non-current due to death) * T (non current due to transfer to another record system, normally another garden) * S (stored in a dormant state) * O (other accession status - different from those above) |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |DonorCategory |E |A code to indicate the type of the immediate donor from which the accession was obtained. This may not necessarily be the original collector of wild material. One or more of the following characters: E, G, B, R, S, U, H, M, N, I, O, U. For a description of these values, please refer to https://www.bgci.org/files/Databases/itf2.pdf, section E.5 (donor type flag). Gardens4Science data providers should also see the note below. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |HybridFlag | * no hybrid: <empty> * named hybrid: * * graft chimaera: + * any other type of hybrid: "hybrid" |Indicates the type of hybrid (if appropriate). |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} '''Note for Gardens4Science data providers:''' Whether or not a given record will be visible in the public view of the [http://gardens4science.biocase.org Gardens4Science data portal] (that is, without being logged in) is determined by the value of the DonorCategory field: Records "O = other" will be visible only for logged-in curators; if the field is filled with "I = individual" or empty, the record will also show up for unregistered users. ===SpecimenUnit/Accessions=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionCatalogue | |Identifier of an accessions catalogue or numbering scheme |- |AccessionDate |2007-07-07 |Date on which the accession was recorded |- |AccessionNumber |001010680 |Institutional number applied to an accession and recorded in the accessions register. |- |} ===Multimedia=== For providing links to associated images, videos and sounds, the sub-tree ''MultimediaObjects'' can be used. Each ABCD unit can have multiple multimedia objects, each represented by its own ''MultimediaObject'' with the following elements: {| class="wikitable" !ABCD element !Example !Remark |- |FileURI* |http://ww2.bgbm.org/herbarium/images/B/-W/14/61/B_-W_14616%20-00%200.jpg |The Universal Resource Identifier (URI) of the underlying multimedia file. |- |ProductURI |http://ww2.bgbm.org/herbarium/view_biocase.cfm?SpecimenPK=726 |The Universal Resource Identifier of an electronic source of a multimedia object including HTML, Javascript or similar means delivering rendered output. |- |Comment |Image of herbarium specimen E00732880 by Specimen Digitisation Pipeline. |Short description of the multimedia object. |- |Format |image/jpeg |A code or descriptive term for the file or object format. |- |FileSize |2365 |The size in Kb of the file referred to by the URI or filename for the multimedia object. |- |Creator |Paris, Museum National d'Histoire Naturelle, OD |The identifier (e.g. name or other id) of personnel who has taken the image. |- |CreatedDate |2012-07-07 |The date the multimedia object was created. |- |ID |22658 |Identifier for the multimedia object relating to or forming the unit. This may be a filename. |- |IPR | |Intellectual Property Rights relating specifically to the multimedia object. This subtree has the same elements as ''Metadata/IPRStatements''. Fore more details, see the respective [[ABCD2Concepts#General_metadata | remarks]]. |- |} 6df35e715bb0aa10f86fc8b29501f269079fc15c 1109 1108 2021-02-01T10:37:15Z JoergHoletschek 3 /* Botanical Garden Unit */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens/observations. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact* |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact* |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData* |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation* |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} '''Note:''' If you publish your data to GBIF, you are required to specify a license in the ''License'' node with the ''language'' attribute and the elements ''Text'' and ''URI''. The latter should contain a link to a well-defined license, for example https://creativecommons.org/publicdomain/zero/1.0/. More details on GBIF licensing can be found [http://www.gbif.org/terms/licences here]. ==Specimen Data== In ACBD terminology, a single specimen/observation is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |LivingSpecimen |An indication of what the unit record describes; needs to comply with a [https://dwc.tdwg.org/terms/#basisOfRecord controlled vocabulary]. |- |KindOfUnit |Seed; (whole) plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="9" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |ScientificName |FullScientificNameString* |Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |Complete name of the taxon including Authors (and years for animals). |- |rowspan="5" | ScientificName/NameAtomised/Botanical |GenusOrMonomial |Aaronsohnia |rowspan="5" | ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |- |FirstEpithet |factorovskyi |- |InfraspecificEpithet |factorovskyi |- |Rank |var. |- |AuthorTeam |Warb. & Eig. |- | - |NameComments | |Free text comments related to the name identified for the unit. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C |The current status of the accession in the garden. Possible values are * C (current accession in the living collection) * D (non-current due to death) * T (non current due to transfer to another record system, normally another garden) * S (stored in a dormant state) * O (other accession status - different from those above) |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |DonorCategory |E |A code to indicate the type of the immediate donor from which the accession was obtained. This may not necessarily be the original collector of wild material. One or more of the following characters: E, G, B, R, S, U, H, M, N, I, O, U. For a description of these values, please refer to https://www.bgci.org/files/Databases/itf2.pdf, section E.5 (donor type flag). Gardens4Science data providers should also see the note below. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |HybridFlag | * |Indicates the type of hybrid (if appropriate), according to * no hybrid: <empty> * named hybrid: * * graft chimaera: + * any other type of hybrid: "hybrid" |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} '''Note for Gardens4Science data providers:''' Whether or not a given record will be visible in the public view of the [http://gardens4science.biocase.org Gardens4Science data portal] (that is, without being logged in) is determined by the value of the DonorCategory field: Records "O = other" will be visible only for logged-in curators; if the field is filled with "I = individual" or empty, the record will also show up for unregistered users. ===SpecimenUnit/Accessions=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionCatalogue | |Identifier of an accessions catalogue or numbering scheme |- |AccessionDate |2007-07-07 |Date on which the accession was recorded |- |AccessionNumber |001010680 |Institutional number applied to an accession and recorded in the accessions register. |- |} ===Multimedia=== For providing links to associated images, videos and sounds, the sub-tree ''MultimediaObjects'' can be used. Each ABCD unit can have multiple multimedia objects, each represented by its own ''MultimediaObject'' with the following elements: {| class="wikitable" !ABCD element !Example !Remark |- |FileURI* |http://ww2.bgbm.org/herbarium/images/B/-W/14/61/B_-W_14616%20-00%200.jpg |The Universal Resource Identifier (URI) of the underlying multimedia file. |- |ProductURI |http://ww2.bgbm.org/herbarium/view_biocase.cfm?SpecimenPK=726 |The Universal Resource Identifier of an electronic source of a multimedia object including HTML, Javascript or similar means delivering rendered output. |- |Comment |Image of herbarium specimen E00732880 by Specimen Digitisation Pipeline. |Short description of the multimedia object. |- |Format |image/jpeg |A code or descriptive term for the file or object format. |- |FileSize |2365 |The size in Kb of the file referred to by the URI or filename for the multimedia object. |- |Creator |Paris, Museum National d'Histoire Naturelle, OD |The identifier (e.g. name or other id) of personnel who has taken the image. |- |CreatedDate |2012-07-07 |The date the multimedia object was created. |- |ID |22658 |Identifier for the multimedia object relating to or forming the unit. This may be a filename. |- |IPR | |Intellectual Property Rights relating specifically to the multimedia object. This subtree has the same elements as ''Metadata/IPRStatements''. Fore more details, see the respective [[ABCD2Concepts#General_metadata | remarks]]. |- |} 82257609891f745ed5914de9480565e03b09032c 1110 1109 2021-02-01T10:39:14Z JoergHoletschek 3 /* Botanical Garden Unit */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens/observations. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact* |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact* |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData* |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation* |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} '''Note:''' If you publish your data to GBIF, you are required to specify a license in the ''License'' node with the ''language'' attribute and the elements ''Text'' and ''URI''. The latter should contain a link to a well-defined license, for example https://creativecommons.org/publicdomain/zero/1.0/. More details on GBIF licensing can be found [http://www.gbif.org/terms/licences here]. ==Specimen Data== In ACBD terminology, a single specimen/observation is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |LivingSpecimen |An indication of what the unit record describes; needs to comply with a [https://dwc.tdwg.org/terms/#basisOfRecord controlled vocabulary]. |- |KindOfUnit |Seed; (whole) plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="9" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |ScientificName |FullScientificNameString* |Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |Complete name of the taxon including Authors (and years for animals). |- |rowspan="5" | ScientificName/NameAtomised/Botanical |GenusOrMonomial |Aaronsohnia |rowspan="5" | ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |- |FirstEpithet |factorovskyi |- |InfraspecificEpithet |factorovskyi |- |Rank |var. |- |AuthorTeam |Warb. & Eig. |- | - |NameComments | |Free text comments related to the name identified for the unit. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C |The current status of the accession in the garden. Possible values are * C (current accession in the living collection) * D (non-current due to death) * T (non current due to transfer to another record system, normally another garden) * S (stored in a dormant state) * O (other accession status - different from those above) |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |DonorCategory |E |A code to indicate the type of the immediate donor from which the accession was obtained. This may not necessarily be the original collector of wild material. One or more of the following characters: E, G, B, R, S, U, H, M, N, I, O, U. For a description of these values, please refer to https://www.bgci.org/files/Databases/itf2.pdf, section E.5 (donor type flag). Gardens4Science data providers should also see the note below. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |HybridFlag | * |Indicates the type of hybrid (if appropriate), according to * no hybrid: <empty> * named hybrid: * (Unicode U+00D7) * graft chimaera: + * any other type of hybrid: "hybrid" |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} '''Note for Gardens4Science data providers:''' Whether or not a given record will be visible in the public view of the [http://gardens4science.biocase.org Gardens4Science data portal] (that is, without being logged in) is determined by the value of the DonorCategory field: Records "O = other" will be visible only for logged-in curators; if the field is filled with "I = individual" or empty, the record will also show up for unregistered users. ===SpecimenUnit/Accessions=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionCatalogue | |Identifier of an accessions catalogue or numbering scheme |- |AccessionDate |2007-07-07 |Date on which the accession was recorded |- |AccessionNumber |001010680 |Institutional number applied to an accession and recorded in the accessions register. |- |} ===Multimedia=== For providing links to associated images, videos and sounds, the sub-tree ''MultimediaObjects'' can be used. Each ABCD unit can have multiple multimedia objects, each represented by its own ''MultimediaObject'' with the following elements: {| class="wikitable" !ABCD element !Example !Remark |- |FileURI* |http://ww2.bgbm.org/herbarium/images/B/-W/14/61/B_-W_14616%20-00%200.jpg |The Universal Resource Identifier (URI) of the underlying multimedia file. |- |ProductURI |http://ww2.bgbm.org/herbarium/view_biocase.cfm?SpecimenPK=726 |The Universal Resource Identifier of an electronic source of a multimedia object including HTML, Javascript or similar means delivering rendered output. |- |Comment |Image of herbarium specimen E00732880 by Specimen Digitisation Pipeline. |Short description of the multimedia object. |- |Format |image/jpeg |A code or descriptive term for the file or object format. |- |FileSize |2365 |The size in Kb of the file referred to by the URI or filename for the multimedia object. |- |Creator |Paris, Museum National d'Histoire Naturelle, OD |The identifier (e.g. name or other id) of personnel who has taken the image. |- |CreatedDate |2012-07-07 |The date the multimedia object was created. |- |ID |22658 |Identifier for the multimedia object relating to or forming the unit. This may be a filename. |- |IPR | |Intellectual Property Rights relating specifically to the multimedia object. This subtree has the same elements as ''Metadata/IPRStatements''. Fore more details, see the respective [[ABCD2Concepts#General_metadata | remarks]]. |- |} ca03e18b64e510672be0358f600b808d517f31de 1111 1110 2021-02-05T15:09:36Z JoergHoletschek 3 /* Botanical Garden Unit */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens/observations. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact* |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact* |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData* |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation* |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} '''Note:''' If you publish your data to GBIF, you are required to specify a license in the ''License'' node with the ''language'' attribute and the elements ''Text'' and ''URI''. The latter should contain a link to a well-defined license, for example https://creativecommons.org/publicdomain/zero/1.0/. More details on GBIF licensing can be found [http://www.gbif.org/terms/licences here]. ==Specimen Data== In ACBD terminology, a single specimen/observation is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |LivingSpecimen |An indication of what the unit record describes; needs to comply with a [https://dwc.tdwg.org/terms/#basisOfRecord controlled vocabulary]. |- |KindOfUnit |Seed; (whole) plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="9" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |ScientificName |FullScientificNameString* |Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |Complete name of the taxon including Authors (and years for animals). |- |rowspan="5" | ScientificName/NameAtomised/Botanical |GenusOrMonomial |Aaronsohnia |rowspan="5" | ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |- |FirstEpithet |factorovskyi |- |InfraspecificEpithet |factorovskyi |- |Rank |var. |- |AuthorTeam |Warb. & Eig. |- | - |NameComments | |Free text comments related to the name identified for the unit. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C |The current status of the accession in the garden. Possible values are * C (current accession in the living collection) * D (non-current due to death) * T (non current due to transfer to another record system, normally another garden) * S (stored in a dormant state) * O (other accession status - different from those above) |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |DonorCategory |E |A code to indicate the type of the immediate donor from which the accession was obtained. This may not necessarily be the original collector of wild material. One or more of the following characters: E, G, B, R, S, U, H, M, N, I, O, U. For a description of these values, please refer to https://www.bgci.org/files/Databases/itf2.pdf, section E.5 (donor type flag). Gardens4Science data providers should also see the note below. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |HybridFlag | x |Indicates the type of hybrid (if appropriate), according to * no hybrid: <empty> * named hybrid: x (Unicode U+00D7) * graft chimaera: + * any other type of hybrid: "hybrid" gardens4science providers should also see the nores below. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} '''Note for Gardens4Science data providers:''' Whether or not a given record will be visible in the public view of the [http://gardens4science.biocase.org Gardens4Science data portal] (that is, without being logged in) is determined by the value of the DonorCategory field: Records "O = other" will be visible only for logged-in curators; if the field is filled with "I = individual" or empty, the record will also show up for unregistered users. ===SpecimenUnit/Accessions=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionCatalogue | |Identifier of an accessions catalogue or numbering scheme |- |AccessionDate |2007-07-07 |Date on which the accession was recorded |- |AccessionNumber |001010680 |Institutional number applied to an accession and recorded in the accessions register. |- |} ===Multimedia=== For providing links to associated images, videos and sounds, the sub-tree ''MultimediaObjects'' can be used. Each ABCD unit can have multiple multimedia objects, each represented by its own ''MultimediaObject'' with the following elements: {| class="wikitable" !ABCD element !Example !Remark |- |FileURI* |http://ww2.bgbm.org/herbarium/images/B/-W/14/61/B_-W_14616%20-00%200.jpg |The Universal Resource Identifier (URI) of the underlying multimedia file. |- |ProductURI |http://ww2.bgbm.org/herbarium/view_biocase.cfm?SpecimenPK=726 |The Universal Resource Identifier of an electronic source of a multimedia object including HTML, Javascript or similar means delivering rendered output. |- |Comment |Image of herbarium specimen E00732880 by Specimen Digitisation Pipeline. |Short description of the multimedia object. |- |Format |image/jpeg |A code or descriptive term for the file or object format. |- |FileSize |2365 |The size in Kb of the file referred to by the URI or filename for the multimedia object. |- |Creator |Paris, Museum National d'Histoire Naturelle, OD |The identifier (e.g. name or other id) of personnel who has taken the image. |- |CreatedDate |2012-07-07 |The date the multimedia object was created. |- |ID |22658 |Identifier for the multimedia object relating to or forming the unit. This may be a filename. |- |IPR | |Intellectual Property Rights relating specifically to the multimedia object. This subtree has the same elements as ''Metadata/IPRStatements''. Fore more details, see the respective [[ABCD2Concepts#General_metadata | remarks]]. |- |} a86f602ce0977da4c4c28ca859d7ddaca2f15bdc 1112 1111 2021-02-05T15:14:46Z JoergHoletschek 3 /* Botanical Garden Unit */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens/observations. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact* |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact* |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData* |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation* |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} '''Note:''' If you publish your data to GBIF, you are required to specify a license in the ''License'' node with the ''language'' attribute and the elements ''Text'' and ''URI''. The latter should contain a link to a well-defined license, for example https://creativecommons.org/publicdomain/zero/1.0/. More details on GBIF licensing can be found [http://www.gbif.org/terms/licences here]. ==Specimen Data== In ACBD terminology, a single specimen/observation is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |LivingSpecimen |An indication of what the unit record describes; needs to comply with a [https://dwc.tdwg.org/terms/#basisOfRecord controlled vocabulary]. |- |KindOfUnit |Seed; (whole) plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="9" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |ScientificName |FullScientificNameString* |Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |Complete name of the taxon including Authors (and years for animals). |- |rowspan="5" | ScientificName/NameAtomised/Botanical |GenusOrMonomial |Aaronsohnia |rowspan="5" | ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |- |FirstEpithet |factorovskyi |- |InfraspecificEpithet |factorovskyi |- |Rank |var. |- |AuthorTeam |Warb. & Eig. |- | - |NameComments | |Free text comments related to the name identified for the unit. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C |The current status of the accession in the garden. Possible values are * C (current accession in the living collection) * D (non-current due to death) * T (non current due to transfer to another record system, normally another garden) * S (stored in a dormant state) * O (other accession status - different from those above) |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |DonorCategory |E |A code to indicate the type of the immediate donor from which the accession was obtained. This may not necessarily be the original collector of wild material. One or more of the following characters: E, G, B, R, S, U, H, M, N, I, O, U. For a description of these values, please refer to https://www.bgci.org/files/Databases/itf2.pdf, section E.5 (donor type flag). Gardens4Science data providers should also see the note below. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |HybridFlag | x |Indicates the type of hybrid (if appropriate), according to * no hybrid: <empty> * named hybrid: x (Unicode U+00D7) * graft chimaera: + * any other type of hybrid: "hybrid" gardens4science providers should also see the notes below. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} '''Note for gardens4Science data providers:''' # Whether or not a given record will be visible in the public view of the [http://gardens4science.biocase.org gardens4science data portal] (that is, without being logged in) is determined by the value of the DonorCategory field: Records "O = other" will be visible only for logged-in curators; if the field is filled with "I = individual" or empty, the record will also show up for unregistered users. # Use''HybridFlag'' in addition to the correct [https://www.bgci.org/files/Databases/itf2.pdf nomenclature of hybrid or graft chimera names] in ''fullScientificName'' (e.g. ''xDisoselenicereus fulgidus (Hook.f.) E.Meier''; ''Epiphyllum x floribundum Kimnach''; ''+Crataegomespilus potsdamiensis''; ''Echinopsis hybrid''). The https://terms.tdwg.org/wiki/abcd2:TaxonIdentified-HybridFlag-@insertionpoint does not need to be mapped/filled and can be ignored (value will be taken from fullScientificName). ===SpecimenUnit/Accessions=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionCatalogue | |Identifier of an accessions catalogue or numbering scheme |- |AccessionDate |2007-07-07 |Date on which the accession was recorded |- |AccessionNumber |001010680 |Institutional number applied to an accession and recorded in the accessions register. |- |} ===Multimedia=== For providing links to associated images, videos and sounds, the sub-tree ''MultimediaObjects'' can be used. Each ABCD unit can have multiple multimedia objects, each represented by its own ''MultimediaObject'' with the following elements: {| class="wikitable" !ABCD element !Example !Remark |- |FileURI* |http://ww2.bgbm.org/herbarium/images/B/-W/14/61/B_-W_14616%20-00%200.jpg |The Universal Resource Identifier (URI) of the underlying multimedia file. |- |ProductURI |http://ww2.bgbm.org/herbarium/view_biocase.cfm?SpecimenPK=726 |The Universal Resource Identifier of an electronic source of a multimedia object including HTML, Javascript or similar means delivering rendered output. |- |Comment |Image of herbarium specimen E00732880 by Specimen Digitisation Pipeline. |Short description of the multimedia object. |- |Format |image/jpeg |A code or descriptive term for the file or object format. |- |FileSize |2365 |The size in Kb of the file referred to by the URI or filename for the multimedia object. |- |Creator |Paris, Museum National d'Histoire Naturelle, OD |The identifier (e.g. name or other id) of personnel who has taken the image. |- |CreatedDate |2012-07-07 |The date the multimedia object was created. |- |ID |22658 |Identifier for the multimedia object relating to or forming the unit. This may be a filename. |- |IPR | |Intellectual Property Rights relating specifically to the multimedia object. This subtree has the same elements as ''Metadata/IPRStatements''. Fore more details, see the respective [[ABCD2Concepts#General_metadata | remarks]]. |- |} 30f16ef7e4ea31a1f7aae1eeb876f438cd9464a6 1113 1112 2021-02-05T15:15:30Z JoergHoletschek 3 /* Botanical Garden Unit */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens/observations. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact* |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact* |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData* |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation* |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} '''Note:''' If you publish your data to GBIF, you are required to specify a license in the ''License'' node with the ''language'' attribute and the elements ''Text'' and ''URI''. The latter should contain a link to a well-defined license, for example https://creativecommons.org/publicdomain/zero/1.0/. More details on GBIF licensing can be found [http://www.gbif.org/terms/licences here]. ==Specimen Data== In ACBD terminology, a single specimen/observation is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |LivingSpecimen |An indication of what the unit record describes; needs to comply with a [https://dwc.tdwg.org/terms/#basisOfRecord controlled vocabulary]. |- |KindOfUnit |Seed; (whole) plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="9" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |ScientificName |FullScientificNameString* |Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |Complete name of the taxon including Authors (and years for animals). |- |rowspan="5" | ScientificName/NameAtomised/Botanical |GenusOrMonomial |Aaronsohnia |rowspan="5" | ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |- |FirstEpithet |factorovskyi |- |InfraspecificEpithet |factorovskyi |- |Rank |var. |- |AuthorTeam |Warb. & Eig. |- | - |NameComments | |Free text comments related to the name identified for the unit. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C |The current status of the accession in the garden. Possible values are * C (current accession in the living collection) * D (non-current due to death) * T (non current due to transfer to another record system, normally another garden) * S (stored in a dormant state) * O (other accession status - different from those above) |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |DonorCategory |E |A code to indicate the type of the immediate donor from which the accession was obtained. This may not necessarily be the original collector of wild material. One or more of the following characters: E, G, B, R, S, U, H, M, N, I, O, U. For a description of these values, please refer to https://www.bgci.org/files/Databases/itf2.pdf, section E.5 (donor type flag). Gardens4Science data providers should also see the note below. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |HybridFlag | x |Indicates the type of hybrid (if appropriate), according to * no hybrid: <empty> * named hybrid: x (Unicode U+00D7) * graft chimaera: + * any other type of hybrid: "hybrid" gardens4science providers should also see the notes below. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} '''Note for gardens4Science data providers:''' # Whether or not a given record will be visible in the public view of the [http://gardens4science.biocase.org gardens4science data portal] (that is, without being logged in) is determined by the value of the ''DonorCategory'' field: Records "O = other" will be visible only for logged-in curators; if the field is filled with "I = individual" or empty, the record will also show up for unregistered users. # Use''HybridFlag'' in addition to the correct [https://www.bgci.org/files/Databases/itf2.pdf nomenclature of hybrid or graft chimera names] in ''fullScientificName'' (e.g. ''xDisoselenicereus fulgidus (Hook.f.) E.Meier''; ''Epiphyllum x floribundum Kimnach''; ''+Crataegomespilus potsdamiensis''; ''Echinopsis hybrid''). The https://terms.tdwg.org/wiki/abcd2:TaxonIdentified-HybridFlag-@insertionpoint does not need to be mapped/filled and can be ignored (value will be taken from fullScientificName). ===SpecimenUnit/Accessions=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionCatalogue | |Identifier of an accessions catalogue or numbering scheme |- |AccessionDate |2007-07-07 |Date on which the accession was recorded |- |AccessionNumber |001010680 |Institutional number applied to an accession and recorded in the accessions register. |- |} ===Multimedia=== For providing links to associated images, videos and sounds, the sub-tree ''MultimediaObjects'' can be used. Each ABCD unit can have multiple multimedia objects, each represented by its own ''MultimediaObject'' with the following elements: {| class="wikitable" !ABCD element !Example !Remark |- |FileURI* |http://ww2.bgbm.org/herbarium/images/B/-W/14/61/B_-W_14616%20-00%200.jpg |The Universal Resource Identifier (URI) of the underlying multimedia file. |- |ProductURI |http://ww2.bgbm.org/herbarium/view_biocase.cfm?SpecimenPK=726 |The Universal Resource Identifier of an electronic source of a multimedia object including HTML, Javascript or similar means delivering rendered output. |- |Comment |Image of herbarium specimen E00732880 by Specimen Digitisation Pipeline. |Short description of the multimedia object. |- |Format |image/jpeg |A code or descriptive term for the file or object format. |- |FileSize |2365 |The size in Kb of the file referred to by the URI or filename for the multimedia object. |- |Creator |Paris, Museum National d'Histoire Naturelle, OD |The identifier (e.g. name or other id) of personnel who has taken the image. |- |CreatedDate |2012-07-07 |The date the multimedia object was created. |- |ID |22658 |Identifier for the multimedia object relating to or forming the unit. This may be a filename. |- |IPR | |Intellectual Property Rights relating specifically to the multimedia object. This subtree has the same elements as ''Metadata/IPRStatements''. Fore more details, see the respective [[ABCD2Concepts#General_metadata | remarks]]. |- |} acd170ee4e0e4cd96d10b8047d690b00021e4a27 1114 1113 2021-02-05T15:15:46Z JoergHoletschek 3 /* Botanical Garden Unit */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens/observations. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact* |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact* |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData* |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation* |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} '''Note:''' If you publish your data to GBIF, you are required to specify a license in the ''License'' node with the ''language'' attribute and the elements ''Text'' and ''URI''. The latter should contain a link to a well-defined license, for example https://creativecommons.org/publicdomain/zero/1.0/. More details on GBIF licensing can be found [http://www.gbif.org/terms/licences here]. ==Specimen Data== In ACBD terminology, a single specimen/observation is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |LivingSpecimen |An indication of what the unit record describes; needs to comply with a [https://dwc.tdwg.org/terms/#basisOfRecord controlled vocabulary]. |- |KindOfUnit |Seed; (whole) plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="9" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |ScientificName |FullScientificNameString* |Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |Complete name of the taxon including Authors (and years for animals). |- |rowspan="5" | ScientificName/NameAtomised/Botanical |GenusOrMonomial |Aaronsohnia |rowspan="5" | ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |- |FirstEpithet |factorovskyi |- |InfraspecificEpithet |factorovskyi |- |Rank |var. |- |AuthorTeam |Warb. & Eig. |- | - |NameComments | |Free text comments related to the name identified for the unit. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C |The current status of the accession in the garden. Possible values are * C (current accession in the living collection) * D (non-current due to death) * T (non current due to transfer to another record system, normally another garden) * S (stored in a dormant state) * O (other accession status - different from those above) |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |DonorCategory |E |A code to indicate the type of the immediate donor from which the accession was obtained. This may not necessarily be the original collector of wild material. One or more of the following characters: E, G, B, R, S, U, H, M, N, I, O, U. For a description of these values, please refer to https://www.bgci.org/files/Databases/itf2.pdf, section E.5 (donor type flag). Gardens4Science data providers should also see the note below. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |HybridFlag | x |Indicates the type of hybrid (if appropriate), according to * no hybrid: <empty> * named hybrid: x (Unicode U+00D7) * graft chimaera: + * any other type of hybrid: "hybrid" gardens4science providers should also see the notes below. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} '''Note for gardens4Science data providers:''' # Whether or not a given record will be visible in the public view of the [http://gardens4science.biocase.org gardens4science data portal] (that is, without being logged in) is determined by the value of the ''DonorCategory'' field: Records "O = other" will be visible only for logged-in curators; if the field is filled with "I = individual" or empty, the record will also show up for unregistered users. # Use ''HybridFlag'' in addition to the correct [https://www.bgci.org/files/Databases/itf2.pdf nomenclature of hybrid or graft chimera names] in ''fullScientificName'' (e.g. ''xDisoselenicereus fulgidus (Hook.f.) E.Meier''; ''Epiphyllum x floribundum Kimnach''; ''+Crataegomespilus potsdamiensis''; ''Echinopsis hybrid''). The https://terms.tdwg.org/wiki/abcd2:TaxonIdentified-HybridFlag-@insertionpoint does not need to be mapped/filled and can be ignored (value will be taken from fullScientificName). ===SpecimenUnit/Accessions=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionCatalogue | |Identifier of an accessions catalogue or numbering scheme |- |AccessionDate |2007-07-07 |Date on which the accession was recorded |- |AccessionNumber |001010680 |Institutional number applied to an accession and recorded in the accessions register. |- |} ===Multimedia=== For providing links to associated images, videos and sounds, the sub-tree ''MultimediaObjects'' can be used. Each ABCD unit can have multiple multimedia objects, each represented by its own ''MultimediaObject'' with the following elements: {| class="wikitable" !ABCD element !Example !Remark |- |FileURI* |http://ww2.bgbm.org/herbarium/images/B/-W/14/61/B_-W_14616%20-00%200.jpg |The Universal Resource Identifier (URI) of the underlying multimedia file. |- |ProductURI |http://ww2.bgbm.org/herbarium/view_biocase.cfm?SpecimenPK=726 |The Universal Resource Identifier of an electronic source of a multimedia object including HTML, Javascript or similar means delivering rendered output. |- |Comment |Image of herbarium specimen E00732880 by Specimen Digitisation Pipeline. |Short description of the multimedia object. |- |Format |image/jpeg |A code or descriptive term for the file or object format. |- |FileSize |2365 |The size in Kb of the file referred to by the URI or filename for the multimedia object. |- |Creator |Paris, Museum National d'Histoire Naturelle, OD |The identifier (e.g. name or other id) of personnel who has taken the image. |- |CreatedDate |2012-07-07 |The date the multimedia object was created. |- |ID |22658 |Identifier for the multimedia object relating to or forming the unit. This may be a filename. |- |IPR | |Intellectual Property Rights relating specifically to the multimedia object. This subtree has the same elements as ''Metadata/IPRStatements''. Fore more details, see the respective [[ABCD2Concepts#General_metadata | remarks]]. |- |} 47c3eaf6e00ff76438b94d33ef73aff4a23d3797 1115 1114 2021-02-05T15:16:33Z JoergHoletschek 3 /* Botanical Garden Unit */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens/observations. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact* |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact* |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData* |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation* |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} '''Note:''' If you publish your data to GBIF, you are required to specify a license in the ''License'' node with the ''language'' attribute and the elements ''Text'' and ''URI''. The latter should contain a link to a well-defined license, for example https://creativecommons.org/publicdomain/zero/1.0/. More details on GBIF licensing can be found [http://www.gbif.org/terms/licences here]. ==Specimen Data== In ACBD terminology, a single specimen/observation is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |LivingSpecimen |An indication of what the unit record describes; needs to comply with a [https://dwc.tdwg.org/terms/#basisOfRecord controlled vocabulary]. |- |KindOfUnit |Seed; (whole) plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="9" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |ScientificName |FullScientificNameString* |Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |Complete name of the taxon including Authors (and years for animals). |- |rowspan="5" | ScientificName/NameAtomised/Botanical |GenusOrMonomial |Aaronsohnia |rowspan="5" | ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |- |FirstEpithet |factorovskyi |- |InfraspecificEpithet |factorovskyi |- |Rank |var. |- |AuthorTeam |Warb. & Eig. |- | - |NameComments | |Free text comments related to the name identified for the unit. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C |The current status of the accession in the garden. Possible values are * C (current accession in the living collection) * D (non-current due to death) * T (non current due to transfer to another record system, normally another garden) * S (stored in a dormant state) * O (other accession status - different from those above) |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |DonorCategory |E |A code to indicate the type of the immediate donor from which the accession was obtained. This may not necessarily be the original collector of wild material. One or more of the following characters: E, G, B, R, S, U, H, M, N, I, O, U. For a description of these values, please refer to https://www.bgci.org/files/Databases/itf2.pdf, section E.5 (donor type flag). Gardens4Science data providers should also see the note below. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |HybridFlag | x |Indicates the type of hybrid (if appropriate), according to * no hybrid: <empty> * named hybrid: x (Unicode U+00D7) * graft chimaera: + * any other type of hybrid: "hybrid" gardens4science providers should also see the notes below. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} '''Note for gardens4Science data providers:''' # Whether or not a given record will be visible in the public view of the [http://gardens4science.biocase.org gardens4science data portal] (that is, without being logged in) is determined by the value of the ''DonorCategory'' field: Records "O = other" will be visible only for logged-in curators; if the field is filled with "I = individual" or empty, the record will also show up for unregistered users. # Use ''HybridFlag'' in addition to the correct [https://www.bgci.org/files/Databases/itf2.pdf nomenclature of hybrid or graft chimera names] in ''fullScientificName'' (e.g. ''xDisoselenicereus fulgidus (Hook.f.) E.Meier''; ''Epiphyllum x floribundum Kimnach''; ''+Crataegomespilus potsdamiensis''; ''Echinopsis hybrid''). The https://terms.tdwg.org/wiki/abcd2:TaxonIdentified-HybridFlag-@insertionpoint does not need to be mapped/filled and can be ignored (value will be taken from ''fullScientificName''). ===SpecimenUnit/Accessions=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionCatalogue | |Identifier of an accessions catalogue or numbering scheme |- |AccessionDate |2007-07-07 |Date on which the accession was recorded |- |AccessionNumber |001010680 |Institutional number applied to an accession and recorded in the accessions register. |- |} ===Multimedia=== For providing links to associated images, videos and sounds, the sub-tree ''MultimediaObjects'' can be used. Each ABCD unit can have multiple multimedia objects, each represented by its own ''MultimediaObject'' with the following elements: {| class="wikitable" !ABCD element !Example !Remark |- |FileURI* |http://ww2.bgbm.org/herbarium/images/B/-W/14/61/B_-W_14616%20-00%200.jpg |The Universal Resource Identifier (URI) of the underlying multimedia file. |- |ProductURI |http://ww2.bgbm.org/herbarium/view_biocase.cfm?SpecimenPK=726 |The Universal Resource Identifier of an electronic source of a multimedia object including HTML, Javascript or similar means delivering rendered output. |- |Comment |Image of herbarium specimen E00732880 by Specimen Digitisation Pipeline. |Short description of the multimedia object. |- |Format |image/jpeg |A code or descriptive term for the file or object format. |- |FileSize |2365 |The size in Kb of the file referred to by the URI or filename for the multimedia object. |- |Creator |Paris, Museum National d'Histoire Naturelle, OD |The identifier (e.g. name or other id) of personnel who has taken the image. |- |CreatedDate |2012-07-07 |The date the multimedia object was created. |- |ID |22658 |Identifier for the multimedia object relating to or forming the unit. This may be a filename. |- |IPR | |Intellectual Property Rights relating specifically to the multimedia object. This subtree has the same elements as ''Metadata/IPRStatements''. Fore more details, see the respective [[ABCD2Concepts#General_metadata | remarks]]. |- |} ed7d4d08e603eab943347f667b1867707b064143 1116 1115 2021-02-05T15:17:09Z JoergHoletschek 3 /* Botanical Garden Unit */ wikitext text/x-wiki This page provides an overview of the most commonly used ABCD elements with short explanations and sample values. If you're looking for a complete list of ABCD elements, please refer to the full documentation at the [https://terms.tdwg.org/wiki/ABCD_2 ABCD Wiki]. Mandatory fields are marked with an asterisk. If you don't provide them, the element - and potentially non-mandatory sibling/parental elements - will be removed from the ABCD document. ==Metadata== Metadata describe the dataset as whole - as opposed to individual specimens/observations. === General metadata === {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="4" | ContentContact* |Name* |John Doe |rowspan="4" | Contact information for the curator of the dataset. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |rowspan="4" | TechnicalContact* |Name* |John Doe |rowspan="4" | Contact information of the person that is responsible for the BioCASe installation - in most cases, this will be the IT department. |- |Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |- |Email |john.doe@acme.com |- |Phone | +49 30 838 99999 |- |Metadata |IconURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |Link to an image that represents the dataset. |- |rowspan="2" | Metadata/RevisionData* |DateCreated |2018-07-19 |rowspan="2" | Date when the dataset has been first published (DateCreated) and of the last changes/additions/removals (DateModified). |- |DateModified* |2018-09-27 |- |rowspan="3" | Metadata/Version |DateIssued |2018-09-27 |rowspan="3" | If you keep track of dataset versions, the info can be specified in this section. |- |Major* |3 |- |Minor |1 |- |rowspan="4" | Metadata/Description/Representation* |Title* |Herbarium Berolinense |rowspan="4" | Short description of the dataset (preferrably in english), consisting of a title, a 2-3 sentences long description and an optional URL. |- |@language* |en |- |Description |The herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem (herbarium acronym: B) is ... |- |URL |https://www.bgbm.org/en/herbarium |- |rowspan="7" | Metadata/Owners/Owner |Addresses/Address |Botanic Garden and Botanical Museum Berlin. Königin-Luise-Str. 6-8; 14195 Berlin, Germany |rowspan="7" | Institution that owns the dataset. |- |EmailAddresses/EmailAddress |john.doe@acme.com |- |TelephoneNumbers/TelephoneNumber | +49 30 838 99999 |- |URIs/URL |https://www.bgbm.org |- |LogoURI |http://ww3.bgbm.org/providerResources/BGBM.jpg |- |Organisation/Name/Representation/Text* |Botanic Garden and Botanical Museum Berlin-Dahlem |- |Organisation/Name/Representation/Abbrevation |BGBM |} === IPR Statements === The section ''Metadata/IPRStatements'' contains several sub-sections that allow to provide IPR (intellectual property rights) statements. Each sub-section contains the mandatory element ''Text'' for the statement and the optional elements ''Details'' and ''URI'' for delivering more details and/or a link to a web page mit more information. The seven sections are as follows: {| class="wikitable" !Section !Example !Remark |- |Acknowledgements |The project was funded by the German Federal Ministry of Education and Research (BMBF) |A free text acknowledging support for the dataset (e.g. grant money, help, permission to reuse published material, etc.). |- |Citation |Röpert D. (ed.) 2000+ [continuously updated]: Digital specimen images at the Herbarium Berolinense. - Published at http://ww2.bgbm.org/herbarium/default.cfm [accessed dd Mmm yyyy]. |Citation guidline for using the data in publications. |- |Disclaimer |No responsibility is accepted for the accuracy of the information in this database. |A disclaimer statement for the dataset (e. g. concerning responsibility for data quality or legal implications). |- |License |Creative Commons 1.0 Public Domain Dedication waiver |The license for using the data; we suggest one of the Creative Commons licenses that can be found at [https://creativecommons.org/licenses/ here]. |- |TermsOfUse |Textual metadata on specimens from the Herbarium Berolinense (B) are released under the Creative Commons 1.0 Public Domain Dedication waiver. |Terms for using the published data. |- |} '''Note:''' If you publish your data to GBIF, you are required to specify a license in the ''License'' node with the ''language'' attribute and the elements ''Text'' and ''URI''. The latter should contain a link to a well-defined license, for example https://creativecommons.org/publicdomain/zero/1.0/. More details on GBIF licensing can be found [http://www.gbif.org/terms/licences here]. ==Specimen Data== In ACBD terminology, a single specimen/observation is called a ''Unit''; for each specimen, one ''Units/Unit'' will be created. Below you'll find the most commonly used ABCD elements in this sub-tree. ===General unit data=== {| class="wikitable" !ABCD element !Example !Remark |- |SourceID* |Herbarium Berolinense |The name or code of the data source. |- |SourceInstitutionID* |B |The unique identifier (code or name) of the institution holding the original data source. |- |UnitID* |B -W 08738 -01 0 |A unique identifier for the unit record within the data source. |- |UnitGUID |http://herbarium.bgbm.org/object/BW08738010 |A globally unique identifier for the unit record. |- |CollectorsFieldNumber |765/10 |The original number assigned to the unit by the collector or observer. |- |RecordBasis |LivingSpecimen |An indication of what the unit record describes; needs to comply with a [https://dwc.tdwg.org/terms/#basisOfRecord controlled vocabulary]. |- |KindOfUnit |Seed; (whole) plant |Part(s), physical state, or class of materials represented by this specimen. |- |Sex |Male |Sex of the unit. |- |RecordURI |http://herbarium.bgbm.org/object/BW08738010 |The web address of the page where more information on this particular unit record can be found. |- |Notes |Specimen examined in a review of the genera of the tribe Eupatorieae, R. M. King & H. Robinson 1968-1969 |A free text description of the unit. Any notes that do not fit elsewhere. |- |DateLastEdited |2009-030-7 |The date of the creation of or last change to this unit record. |- |LastEditor |John Doe |The name or other indication of the person or team that created or last edited this record. |- |} ===Gathering event=== The ''Gathering'' subsection of ABCD covers all details about the gathering site and the gathering agents. All the follwing groups and elements are located in ''Units/Unit/Gathering''. {| class="wikitable" !Group !Element !Example !Remark |- |rowspan="2" | Agents |GatheringAgentsText |Scholz, Sipman |A free text representation or concatenated version of those responsible for collecting or recording. In case of a gathering team, it is advised not to provide a concatenated text of the collectors, but individual gathering agents (see below). |- |GatheringAgent/AgentText |Sipman |Name of an individual gathering agent. The element ''GatheringAgent'' is repeatable, so it can be provided several times in case of a gathering team. |- |rowspan="4" | Altitude |MeasurementOrFactText |1500-1600m |Use this element if you have the altitude stored as a free text, that is as a single, concatenated text. If the boundaries of a range are stored as separate values in your database, use the ''MeasurementOrFactAtomised'' elements below. |- |MeasurementOrFactAtomised/LowerValue* |1500 |The lower or only value of the gathering site altitude. |- |MeasurementOrFactAtomised/UpperValue |1600 |The upper value for the measured gathering site altitude, if there is a range. |- |MeasurementOrFactAtomised/UnitOfMeasurement |m |The unit of measurement used to measure the gathering site altitude. |- |rowspan="2" | Country |ISO3166Code |US |ISO code of the country where the voucher was collected/observed. |- |Name* |United States of America |Name of the gathering country. |- |rowspan="3" | DateTime |DateText |August 2012 |Date when voucher was collected if it doesn't conform to a specific date format and is stored just as a freet text. |- |ISODateTimeBegin |2012-08-17 |Gathering date in ISO format. If you want to provide a time span, this will be the start of the time span. |- |ISODateTimeEnd |2012-08-19 |End datetime (in ISO format) if the gathering covers a time span. |- | |LocalityText |Landkreis Neustrelitz, Feldberger Seengebiet, Carwitz, ca. 1 km SW von Carwitz. |The original gathering locality data as appearing on a label or in an original entry, as a text string. This field should be transcribed verbatim from the specimen label. |- |rowspan="2" | NamedArea |AreaName |Europe |Any geographic term that is neither a country (use ''Country'' element) nor the exact locality description (use ''LocalityText''). |- |AreaClass |Continent |Type of geographic term provided in ''AreaName'' |- |Biotope |Text |Amongst rocks and stones, calcareous soil and screes |Gathering site biotope description. |- | - |AreaDetail | |A free text description of further details pertaining to the observation, sampling, or collection locality which which did not fit into the one of the other elements. |- |rowspan="8" | SiteCoordinateSets/ SiteCoordinates/ CoordinatesLatLong |../CoordinateMethod |Inferred map reference |Method of coordinates determination (GPS vs. inferred from description). |- |LatitudeDecimal |52.49179 |Latitude in decimal degrees. |- |LongitudeDecimal |13.23589 |Longitude in decimal degreed. |- |CoordinatesText |52°30' N 13°23' E |An alternative concatenated text representation of the lat/lon coordinates if they can't be provided as separate lat/lon values. |- |SpatialDatum |WGS 84 |A mathematical surface on which the mapping and coordinate system used for the geocodes of the gathering site are based. |- |CoordinateErrorDistanceInMeters |500 |An estimate of how tightly the collecting locality was specified, expressed as a distance in meters corresponding to a radius around the lat/long coordinates. |- |AccuracyStatement |Coordinates rounded to 2 decimal places. |A free text statement of the degree of accuracy of the latitude and longitude coordinates. |- |CoordinateErrorMethod |Maximum error defined by dimensions of original ordinance survey map. |Verbatim statement of how the Coordinate Error was determined. |- | - |Notes | |Further text relating to the gathering event that does not fit one of the specified elements. |- |} ===Identification history=== Most specimen databases record the complete determination or identification history of a single specimen. ABCD allows to provide this complete history; each determination becomes an ''Identifications/Identification'' subtree. {| class="wikitable" !Group !Subgroup !Element !Example !Remark |- | - | - |PreferredFlag |1 |The currently preferred determination should be marked with ''1'' or ''True'', historic determinations with ''0'' or ''False''. |- |Date | - |DateText/ISODateTimeBegin |2018-08-17 |Date of the unit identification event. If in ISO format, use ''ISODateTimeBegin'', ''DateText'' otherwise. |- |rowspan="2" | Identifiers | - |IdentifiersText |Humboldt & Bonpland |Text indicating the person, person team, or organisation that made the identification. If it is a team, individual ''Identifier'' subtrees should be provided, if possible (see below). |- |Identifier/PersonName |FullName |Humboldt |Person that made the identification. |- |rowspan="9" | Result/TaxonIdentified |rowspan="2" | HigherTaxa/HigherTaxon |HigherTaxonName* |Asteraceae |Name of the higher taxon. Please have a look at the BioCASE Wiki for how to prepare your database for the [http://wiki.bgbm.org/bps/index.php/Preparation#Repeatable_Elements_in_ABCD repeatable elements] |- |HigherTaxonRank |familia |Rame of the higher taxon. |- |ScientificName |FullScientificNameString* |Aaronsohnia factorovskyi Warb. & Eig. var. factorovskyi |Complete name of the taxon including Authors (and years for animals). |- |rowspan="5" | ScientificName/NameAtomised/Botanical |GenusOrMonomial |Aaronsohnia |rowspan="5" | ABCD has several container for NameAtomised, it depens on your sampes which one to choose (Botanical or Zoology etc.) |- |FirstEpithet |factorovskyi |- |InfraspecificEpithet |factorovskyi |- |Rank |var. |- |AuthorTeam |Warb. & Eig. |- | - |NameComments | |Free text comments related to the name identified for the unit. |- |rowspan="2" | References/Reference |rowspan="2" | - |TitleCitation |Ralfs, J: The British Desmidieae, London 1848 |rowspan="2" | A reference (e.g. an article or monograph) that was used as the base for the identifier's taxon identification (e.g. by means of providing a key or a description of the taxon). |- |CitationDetail |p. 22 + 226, 35 pl. |- | - | - |Notes |Invalid publication as: Echinella rotata Greville in W.J. Hooker 1833: British Flora Vol II, pp. 432, London, p.398 |Supplementary remarks about the unit identification. |- |} ===Botanical Garden Unit=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionStatus |C |The current status of the accession in the garden. Possible values are * C (current accession in the living collection) * D (non-current due to death) * T (non current due to transfer to another record system, normally another garden) * S (stored in a dormant state) * O (other accession status - different from those above) |- |Cultivation | |Allows for cultivation information to be passed as free text in order to help the receiving garden care for the transferred accession or to record the garden's experience made with this accession. |- |DonorCategory |E |A code to indicate the type of the immediate donor from which the accession was obtained. This may not necessarily be the original collector of wild material. One or more of the following characters: E, G, B, R, S, U, H, M, N, I, O, U. For a description of these values, please refer to https://www.bgci.org/files/Databases/itf2.pdf, section E.5 (donor type flag). Gardens4Science data providers should also see the note below. |- |Hardiness |1 |A flag indicating that the accession is grown outdoors in the botanical garden. |- |HybridFlag | x |Indicates the type of hybrid (if appropriate), according to * no hybrid: <empty> * named hybrid: x * graft chimaera: + * any other type of hybrid: "hybrid" gardens4science providers should also see the notes below. |- |IPEN |XX-0-HEID-3253 |A unique number given to plant materials accessioned in conformance with the rules of the International Plant Exchange Network. |- |LocationInGarden |MD4 |The current location of the living plant in the botanical garden. |- |Perennation | |A code to indicate the means of perennation. |- |PlantingDate |2007-07-07 |The date in ISO format the plant was planted at its current location in the botanical garden. |- |} '''Note for gardens4Science data providers:''' # Whether or not a given record will be visible in the public view of the [http://gardens4science.biocase.org gardens4science data portal] (that is, without being logged in) is determined by the value of the ''DonorCategory'' field: Records "O = other" will be visible only for logged-in curators; if the field is filled with "I = individual" or empty, the record will also show up for unregistered users. # Use ''HybridFlag'' in addition to the correct [https://www.bgci.org/files/Databases/itf2.pdf nomenclature of hybrid or graft chimera names] in ''fullScientificName'' (e.g. ''xDisoselenicereus fulgidus (Hook.f.) E.Meier''; ''Epiphyllum x floribundum Kimnach''; ''+Crataegomespilus potsdamiensis''; ''Echinopsis hybrid''). The https://terms.tdwg.org/wiki/abcd2:TaxonIdentified-HybridFlag-@insertionpoint does not need to be mapped/filled and can be ignored (value will be taken from ''fullScientificName''). ===SpecimenUnit/Accessions=== {| class="wikitable" !ABCD element !Example !Remark |- |AccessionCatalogue | |Identifier of an accessions catalogue or numbering scheme |- |AccessionDate |2007-07-07 |Date on which the accession was recorded |- |AccessionNumber |001010680 |Institutional number applied to an accession and recorded in the accessions register. |- |} ===Multimedia=== For providing links to associated images, videos and sounds, the sub-tree ''MultimediaObjects'' can be used. Each ABCD unit can have multiple multimedia objects, each represented by its own ''MultimediaObject'' with the following elements: {| class="wikitable" !ABCD element !Example !Remark |- |FileURI* |http://ww2.bgbm.org/herbarium/images/B/-W/14/61/B_-W_14616%20-00%200.jpg |The Universal Resource Identifier (URI) of the underlying multimedia file. |- |ProductURI |http://ww2.bgbm.org/herbarium/view_biocase.cfm?SpecimenPK=726 |The Universal Resource Identifier of an electronic source of a multimedia object including HTML, Javascript or similar means delivering rendered output. |- |Comment |Image of herbarium specimen E00732880 by Specimen Digitisation Pipeline. |Short description of the multimedia object. |- |Format |image/jpeg |A code or descriptive term for the file or object format. |- |FileSize |2365 |The size in Kb of the file referred to by the URI or filename for the multimedia object. |- |Creator |Paris, Museum National d'Histoire Naturelle, OD |The identifier (e.g. name or other id) of personnel who has taken the image. |- |CreatedDate |2012-07-07 |The date the multimedia object was created. |- |ID |22658 |Identifier for the multimedia object relating to or forming the unit. This may be a filename. |- |IPR | |Intellectual Property Rights relating specifically to the multimedia object. This subtree has the same elements as ''Metadata/IPRStatements''. Fore more details, see the respective [[ABCD2Concepts#General_metadata | remarks]]. |- |} 49d376c3a9bda48d9641f979f78356cf30bb9e3f 0 23 1074 981 2018-10-02T11:23:51Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool c93844b526f9870960b4386e2671ec8bb5886532 1075 1074 2019-01-16T11:55:12Z JoergHoletschek 3 wikitext text/x-wiki == Current Version 3 == === Upcoming === * Bugfix for Local Query Tool (Error page not displayed properly) === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 1f3aaaab02da1ecf0f2a0502db12999541f109d8 1076 1075 2019-01-16T12:36:50Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** Bugfix for SSL handshake error if HTTPS is used. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 3b9f1251d6ec822fedd46980cb9be0d04f222efa 1077 1076 2019-01-16T12:37:29Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** Bugfix for SSL handshake error on certain installations if HTTPS is used. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool c7ac00f73c6a18ca740103677276fe788a20895b 1078 1077 2019-01-16T12:47:30Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** Bugfix for SSL handshake error on certain installations if HTTPS is used. * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 0a0ad5ba3d8a45f24b270ead1f9cb8ec31f4e743 1081 1078 2019-04-16T14:34:14Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** Bugfix for SSL handshake error on certain installations if HTTPS is used. * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 3e4f4bf0548cee91c2c62102cd3304a49187d98c 1086 1081 2019-05-02T10:47:48Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool cd167beb3bc7ca55ce48a3da053107f2059b49bd 1087 1086 2019-05-15T15:34:38Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool bda7721a438d1fc4fe97592895453b4bcf96f9df 1088 1087 2019-07-10T07:37:04Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 1dae9856039d5186c3a9ea5e9b52468aafe14834 1089 1088 2019-07-10T07:38:53Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 51717a3a914e39d1b459750272268b2c675687d6 1090 1089 2019-08-20T10:19:02Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool b513c6811674cc0d402fe6ed44a198259b27fe2c 1091 1090 2019-08-21T08:42:45Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 32c97655e447af2c69b36106040bc447a705122b 1092 1091 2019-09-23T09:44:00Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If colum retrieval is not available, potential dots are removed from column names, since they're not allowed. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool c8f1786a608e78fdc49f59d6ab015e9451acaa84 1093 1092 2019-11-18T11:55:35Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 91f69d3439abe2dc69e018de09bf7db4378ced4f 1094 1093 2019-11-18T15:04:15Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters (namely <>!&/"). === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool aa9ce3eebbc18adc280266e88a0722b6d5456ab2 1095 1094 2019-11-18T15:04:33Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 0bfee144850967dcb9445156592f1598edbd1610 1096 1095 2019-12-02T14:54:22Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. * CGI Parameters sanitised for security reasons. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 7f498b2f3e3939acd2e41755d7706e0fc6e945f2 1097 1096 2019-12-02T14:54:45Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. * CGI Parameters sanitised for security reasons. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool b3aedc36d936fa993932f171ae2e550ac2c91ae7 1098 1097 2019-12-02T15:10:44Z JoergHoletschek 3 /* Version 3.7.3 [2019-12-02] */ wikitext text/x-wiki == Current Version 3 == === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 342e8d61b98259f3267ed710acade462a44e32e2 1103 1098 2020-07-06T15:48:35Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Fix in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 667334e562d6c07bc3aac6c26f0c36f55bf082f0 1104 1103 2020-07-20T16:22:18Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Bugfix for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). * Fix in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool e193bd65e3dacdba74fef5857021dbbd0d153249 1105 1104 2020-11-19T17:45:33Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool b784ff9f0e1ad02c5cfe27253e395144604db7bf 1117 1105 2021-02-10T16:30:28Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 5eb58b8bd905d6ad23d786f6051b38d528cc91b2 1118 1117 2021-05-17T09:44:15Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool a919aad5d9b09da48633910de379aa3b2c82ee3e 0 52 1106 864 2020-11-30T13:03:35Z JoergHoletschek 3 /* Installation */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/ipt) you can create a DarwinCore Archive (see http://rs.tdwg.org/dwc/terms/guides/text/index.htm for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I'm using Internet Information Server, and after creating the Virtual Directory for BioCASe, I get a “502.2 - Bad Gateway. The specified CGI application misbehaved by not returning a complete set of HTTP headers” error.=== This happens when you didn't correctly associate .cgi and .py files with Python. Make sure you've linked both files with Python and that the path contains the placeholders ''%s %s'' (see [[Installation#Internet_Information_Server|Linking IIS]]). ===I cannot find the installation package of the database connectivity module for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later of psycopg2. ===I'm trying to use BioCASe with Postgres; for this, I tried to install psycopg on Windows. But it doesn't work - either the install crashes, or it finishes, but the test libs page still reports psycopg2 as not installed. In the debug log I can find the error message "ImportError: DLL load failed (...)". What's wrong?=== We suspect this is caused by a bug in recent psycopg2 releases (find more [http://psycopg.lighthouseapp.com/projects/62710/tickets/20 here]). On our test installations, we managed to get round this issue by installing pycopg2 with administration rights. Do do this, right-click the installer file and choose "Run as administrator". ===What's the difference between the DBMS packages odbc_tsql and odbc_tsql_2012 and between pymssql/pymssql_2012, respectively?=== The dbmods ending in "2012" use the new OFFSET clause that was introduced in MS SQL Server 2012. This will make paging faster and increase performance for larger datasets. They work only with SQL Server 2012 or later, and if you use SQL Server 2012, you should use them for best performance. Dbmods without a trailing "2012" work for both older versions and SQL Server 2012, but will be slower in paging large datasets. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== See [[Installation#Updating_an_existing_BioCASe_Installation|Updating an existing BioCASe Installation]]. ===I've updated my installation from a version prior to 2.6. Now, when I enter the DB structure, some of the table and column names are gone, the same happens in the mapping editor.=== Starting with version 2.6, BioCASe retrieves table and column names from the database for some database management systems (namely MySQL, SQL Server, Postgres, Foxpro, Access and Oracle). In older versions, you had to enter table/column names manually, and when you did the setup with such an early version, you probably didn't care for upper/lower cases. Since some DB management systems are case-sensitive, BioCASe can't ignore the differences and won't find these technically incorrect names in the lists retrieved from the database. The respective drop down boxes will therefore be empty. If that happens, just select the correct names from the drop down lists. ===When I try to create an archive, I get a "Permission denied" error. Why is that?=== For creating archives, you need to grant write permissions for the folders ''archive'' and ''www/downloads'' of your BioCASe installation to Python. You can check if permissions are set correctly in the test libs page (Start --> Utilities --> Test Libs). ===I'm trying to install BioCASe on Ubuntu (or another Linux). Even though I've run the setup script and added the snippet to the Apache configuration file, I see source code when I try out my installation instead of the pages described in the documentation=== Apparently CGI is not enabled on your machine. Google how to enable CGI for your distribution and follow the instructions you'll find. ===When trying to run BioCASe after installation, I get an internal server error (500). But the log folder of BioCASe is empty, nothing to find.=== This is probably because BioCASe - or the web server that runs BioCASe - can't write to this folder. Make sure that the web server (user ''www-data'' for Apache) has write permissions for this folder. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other only by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===I've mapped ABCD2. When testing the web service, I get a negative number for "RecordsDropped" in the response document. What does that mean?=== Same problem as above - it means you have an error in your mapping. Most probably you've mapped a non-repeatable ABCD concept to a table/column in your database that holds several entries per unit (occurrence) record. The BioCASe Provider Software will duplicate the respective unit records to be able to create valid ABCD documents - which is indicated by a negative number of "records dropped". Make sure the cardinality of your database relations match the structure (in respects of element repeatability) of ABCD to avoid this problem. ===I'm using pymssql to connect to a SQL Server database. Char(n) and Varchar(n) strings get truncated to 255 characters - what's wrong?=== This is known limitation of TDS protocol used by pymssql. Just change the data type for columns storing long string to TEXT or create a view that converts Char/Varchar columns to TEXT columns on the fly to work around this issue. ===I'm using pymssql to connect to a SQL Server database, BioCASe is running on a Linux distribution. When I do a Search or Scan request, I don't get any results (0 hits), even though there are definitely records in the database.=== That problem occurs on old versions of pymssql (1.0.2 or earlier) on Linux distributions. Update to version 2.0 or later to solve this. If you still get empty result sets, these posts might be helpful: * http://open-up.cybertaxonomy.africamuseum.be/forum_topic/important_if_you_use_biocase_bps_linux_platform_connect_ms_sql_server * http://code.google.com/p/pymssql/issues/detail?id=76. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===I am using BioCASe on a Windows Server. For some queries with the Local QueryForm, I get an error page that is obviously produced by the web server and not BioCASe. It reads "Error Code: 500 Internal Server Error. The request was rejected by the HTTP filter. Contact the server administrator. (12217)"=== This happens when BioCASe is run behind a Microsoft ISA Server/Microsoft Forefront Threat Management Gateway. To solve this problem, follow the instructions given in this article: http://support.microsoft.com/kb/837865. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. ===I'm using BioCASe in a Vagrant Virtual Box on MacOS with Synced Folders. When using the Manual Query Form or the Archiving page, I get an "ImportError: No module named wrapper.errorclasses."=== We don't know exactly what causes this, but apparently it happens when BioCASe is run in a Vagrant Virtual Box on MacOS and Vagrant's synced folder are used. We do have a workaround for this, but it is not easy. Please [[Special:Contact|contact us]] for assistance on this. ===I'm using BioCASe on a Windows server and odbc_tsql to connect to an MS SQL Server. Even though I do get results for a SCAN request, I cannot do a SEARCH (invalid document received).=== Please check your encoding settings in the DB connection configuration. If it is set to UTF_8, try Latin_1 instead. == Trouble Shooting == ===I’ve forgotten the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. 91789348c04f6d4044da2b08d963f6e6e7ebdb2d 1119 1106 2021-05-26T08:27:44Z JoergHoletschek 3 /* Installation */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/ipt) you can create a DarwinCore Archive (see http://rs.tdwg.org/dwc/terms/guides/text/index.htm for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===After installing BioCASe, I get a "500: Internal Server Error"=== # Make sure you've run the setup-script after downloading or checking out BioCASe with the right Python version. To check this, open the file ''www/index.cgi'' of your installation in a text editor and check the first line: This is the Python executable used by Python. # Empty the folder ''cache'' of your installation - please remove all folders and files you find here. # If the problem persists, please contact us! ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I'm using Internet Information Server, and after creating the Virtual Directory for BioCASe, I get a “502.2 - Bad Gateway. The specified CGI application misbehaved by not returning a complete set of HTTP headers” error.=== This happens when you didn't correctly associate .cgi and .py files with Python. Make sure you've linked both files with Python and that the path contains the placeholders ''%s %s'' (see [[Installation#Internet_Information_Server|Linking IIS]]). ===I cannot find the installation package of the database connectivity module for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later of psycopg2. ===I'm trying to use BioCASe with Postgres; for this, I tried to install psycopg on Windows. But it doesn't work - either the install crashes, or it finishes, but the test libs page still reports psycopg2 as not installed. In the debug log I can find the error message "ImportError: DLL load failed (...)". What's wrong?=== We suspect this is caused by a bug in recent psycopg2 releases (find more [http://psycopg.lighthouseapp.com/projects/62710/tickets/20 here]). On our test installations, we managed to get round this issue by installing pycopg2 with administration rights. Do do this, right-click the installer file and choose "Run as administrator". ===What's the difference between the DBMS packages odbc_tsql and odbc_tsql_2012 and between pymssql/pymssql_2012, respectively?=== The dbmods ending in "2012" use the new OFFSET clause that was introduced in MS SQL Server 2012. This will make paging faster and increase performance for larger datasets. They work only with SQL Server 2012 or later, and if you use SQL Server 2012, you should use them for best performance. Dbmods without a trailing "2012" work for both older versions and SQL Server 2012, but will be slower in paging large datasets. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== See [[Installation#Updating_an_existing_BioCASe_Installation|Updating an existing BioCASe Installation]]. ===I've updated my installation from a version prior to 2.6. Now, when I enter the DB structure, some of the table and column names are gone, the same happens in the mapping editor.=== Starting with version 2.6, BioCASe retrieves table and column names from the database for some database management systems (namely MySQL, SQL Server, Postgres, Foxpro, Access and Oracle). In older versions, you had to enter table/column names manually, and when you did the setup with such an early version, you probably didn't care for upper/lower cases. Since some DB management systems are case-sensitive, BioCASe can't ignore the differences and won't find these technically incorrect names in the lists retrieved from the database. The respective drop down boxes will therefore be empty. If that happens, just select the correct names from the drop down lists. ===When I try to create an archive, I get a "Permission denied" error. Why is that?=== For creating archives, you need to grant write permissions for the folders ''archive'' and ''www/downloads'' of your BioCASe installation to Python. You can check if permissions are set correctly in the test libs page (Start --> Utilities --> Test Libs). ===I'm trying to install BioCASe on Ubuntu (or another Linux). Even though I've run the setup script and added the snippet to the Apache configuration file, I see source code when I try out my installation instead of the pages described in the documentation=== Apparently CGI is not enabled on your machine. Google how to enable CGI for your distribution and follow the instructions you'll find. ===When trying to run BioCASe after installation, I get an internal server error (500). But the log folder of BioCASe is empty, nothing to find.=== This is probably because BioCASe - or the web server that runs BioCASe - can't write to this folder. Make sure that the web server (user ''www-data'' for Apache) has write permissions for this folder. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other only by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===I've mapped ABCD2. When testing the web service, I get a negative number for "RecordsDropped" in the response document. What does that mean?=== Same problem as above - it means you have an error in your mapping. Most probably you've mapped a non-repeatable ABCD concept to a table/column in your database that holds several entries per unit (occurrence) record. The BioCASe Provider Software will duplicate the respective unit records to be able to create valid ABCD documents - which is indicated by a negative number of "records dropped". Make sure the cardinality of your database relations match the structure (in respects of element repeatability) of ABCD to avoid this problem. ===I'm using pymssql to connect to a SQL Server database. Char(n) and Varchar(n) strings get truncated to 255 characters - what's wrong?=== This is known limitation of TDS protocol used by pymssql. Just change the data type for columns storing long string to TEXT or create a view that converts Char/Varchar columns to TEXT columns on the fly to work around this issue. ===I'm using pymssql to connect to a SQL Server database, BioCASe is running on a Linux distribution. When I do a Search or Scan request, I don't get any results (0 hits), even though there are definitely records in the database.=== That problem occurs on old versions of pymssql (1.0.2 or earlier) on Linux distributions. Update to version 2.0 or later to solve this. If you still get empty result sets, these posts might be helpful: * http://open-up.cybertaxonomy.africamuseum.be/forum_topic/important_if_you_use_biocase_bps_linux_platform_connect_ms_sql_server * http://code.google.com/p/pymssql/issues/detail?id=76. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===I am using BioCASe on a Windows Server. For some queries with the Local QueryForm, I get an error page that is obviously produced by the web server and not BioCASe. It reads "Error Code: 500 Internal Server Error. The request was rejected by the HTTP filter. Contact the server administrator. (12217)"=== This happens when BioCASe is run behind a Microsoft ISA Server/Microsoft Forefront Threat Management Gateway. To solve this problem, follow the instructions given in this article: http://support.microsoft.com/kb/837865. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. ===I'm using BioCASe in a Vagrant Virtual Box on MacOS with Synced Folders. When using the Manual Query Form or the Archiving page, I get an "ImportError: No module named wrapper.errorclasses."=== We don't know exactly what causes this, but apparently it happens when BioCASe is run in a Vagrant Virtual Box on MacOS and Vagrant's synced folder are used. We do have a workaround for this, but it is not easy. Please [[Special:Contact|contact us]] for assistance on this. ===I'm using BioCASe on a Windows server and odbc_tsql to connect to an MS SQL Server. Even though I do get results for a SCAN request, I cannot do a SEARCH (invalid document received).=== Please check your encoding settings in the DB connection configuration. If it is set to UTF_8, try Latin_1 instead. == Trouble Shooting == ===I’ve forgotten the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. 602e6fbd60bca1a00fdcb40e5f3ed9228cf1e997 1120 1119 2021-05-26T08:30:08Z JoergHoletschek 3 /* After installing BioCASe, I get a "500: Internal Server Error" */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/ipt) you can create a DarwinCore Archive (see http://rs.tdwg.org/dwc/terms/guides/text/index.htm for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===After installing BioCASe, I get a "500: Internal Server Error"=== # Make sure you've [[Installation#Running_the_Setup_Script | run the setup script]] after downloading or checking out BioCASe with the right Python version. To check this, open the file ''www/index.cgi'' of your installation in a text editor and check the first line: This is the Python executable used by Python. # Empty the folder ''cache'' of your installation - please remove all folders and files you find here. # If the problem persists, please contact us! ===I already have a newer Python version installed on my server which is already used by another application. I read that Python 2.5 is recommended for BioCASe. What should I do?=== You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install Python 2.5 for BioCASe. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I'm using Internet Information Server, and after creating the Virtual Directory for BioCASe, I get a “502.2 - Bad Gateway. The specified CGI application misbehaved by not returning a complete set of HTTP headers” error.=== This happens when you didn't correctly associate .cgi and .py files with Python. Make sure you've linked both files with Python and that the path contains the placeholders ''%s %s'' (see [[Installation#Internet_Information_Server|Linking IIS]]). ===I cannot find the installation package of the database connectivity module for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later of psycopg2. ===I'm trying to use BioCASe with Postgres; for this, I tried to install psycopg on Windows. But it doesn't work - either the install crashes, or it finishes, but the test libs page still reports psycopg2 as not installed. In the debug log I can find the error message "ImportError: DLL load failed (...)". What's wrong?=== We suspect this is caused by a bug in recent psycopg2 releases (find more [http://psycopg.lighthouseapp.com/projects/62710/tickets/20 here]). On our test installations, we managed to get round this issue by installing pycopg2 with administration rights. Do do this, right-click the installer file and choose "Run as administrator". ===What's the difference between the DBMS packages odbc_tsql and odbc_tsql_2012 and between pymssql/pymssql_2012, respectively?=== The dbmods ending in "2012" use the new OFFSET clause that was introduced in MS SQL Server 2012. This will make paging faster and increase performance for larger datasets. They work only with SQL Server 2012 or later, and if you use SQL Server 2012, you should use them for best performance. Dbmods without a trailing "2012" work for both older versions and SQL Server 2012, but will be slower in paging large datasets. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== See [[Installation#Updating_an_existing_BioCASe_Installation|Updating an existing BioCASe Installation]]. ===I've updated my installation from a version prior to 2.6. Now, when I enter the DB structure, some of the table and column names are gone, the same happens in the mapping editor.=== Starting with version 2.6, BioCASe retrieves table and column names from the database for some database management systems (namely MySQL, SQL Server, Postgres, Foxpro, Access and Oracle). In older versions, you had to enter table/column names manually, and when you did the setup with such an early version, you probably didn't care for upper/lower cases. Since some DB management systems are case-sensitive, BioCASe can't ignore the differences and won't find these technically incorrect names in the lists retrieved from the database. The respective drop down boxes will therefore be empty. If that happens, just select the correct names from the drop down lists. ===When I try to create an archive, I get a "Permission denied" error. Why is that?=== For creating archives, you need to grant write permissions for the folders ''archive'' and ''www/downloads'' of your BioCASe installation to Python. You can check if permissions are set correctly in the test libs page (Start --> Utilities --> Test Libs). ===I'm trying to install BioCASe on Ubuntu (or another Linux). Even though I've run the setup script and added the snippet to the Apache configuration file, I see source code when I try out my installation instead of the pages described in the documentation=== Apparently CGI is not enabled on your machine. Google how to enable CGI for your distribution and follow the instructions you'll find. ===When trying to run BioCASe after installation, I get an internal server error (500). But the log folder of BioCASe is empty, nothing to find.=== This is probably because BioCASe - or the web server that runs BioCASe - can't write to this folder. Make sure that the web server (user ''www-data'' for Apache) has write permissions for this folder. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other only by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===I've mapped ABCD2. When testing the web service, I get a negative number for "RecordsDropped" in the response document. What does that mean?=== Same problem as above - it means you have an error in your mapping. Most probably you've mapped a non-repeatable ABCD concept to a table/column in your database that holds several entries per unit (occurrence) record. The BioCASe Provider Software will duplicate the respective unit records to be able to create valid ABCD documents - which is indicated by a negative number of "records dropped". Make sure the cardinality of your database relations match the structure (in respects of element repeatability) of ABCD to avoid this problem. ===I'm using pymssql to connect to a SQL Server database. Char(n) and Varchar(n) strings get truncated to 255 characters - what's wrong?=== This is known limitation of TDS protocol used by pymssql. Just change the data type for columns storing long string to TEXT or create a view that converts Char/Varchar columns to TEXT columns on the fly to work around this issue. ===I'm using pymssql to connect to a SQL Server database, BioCASe is running on a Linux distribution. When I do a Search or Scan request, I don't get any results (0 hits), even though there are definitely records in the database.=== That problem occurs on old versions of pymssql (1.0.2 or earlier) on Linux distributions. Update to version 2.0 or later to solve this. If you still get empty result sets, these posts might be helpful: * http://open-up.cybertaxonomy.africamuseum.be/forum_topic/important_if_you_use_biocase_bps_linux_platform_connect_ms_sql_server * http://code.google.com/p/pymssql/issues/detail?id=76. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===I am using BioCASe on a Windows Server. For some queries with the Local QueryForm, I get an error page that is obviously produced by the web server and not BioCASe. It reads "Error Code: 500 Internal Server Error. The request was rejected by the HTTP filter. Contact the server administrator. (12217)"=== This happens when BioCASe is run behind a Microsoft ISA Server/Microsoft Forefront Threat Management Gateway. To solve this problem, follow the instructions given in this article: http://support.microsoft.com/kb/837865. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. ===I'm using BioCASe in a Vagrant Virtual Box on MacOS with Synced Folders. When using the Manual Query Form or the Archiving page, I get an "ImportError: No module named wrapper.errorclasses."=== We don't know exactly what causes this, but apparently it happens when BioCASe is run in a Vagrant Virtual Box on MacOS and Vagrant's synced folder are used. We do have a workaround for this, but it is not easy. Please [[Special:Contact|contact us]] for assistance on this. ===I'm using BioCASe on a Windows server and odbc_tsql to connect to an MS SQL Server. Even though I do get results for a SCAN request, I cannot do a SEARCH (invalid document received).=== Please check your encoding settings in the DB connection configuration. If it is set to UTF_8, try Latin_1 instead. == Trouble Shooting == ===I’ve forgotten the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. b0d658893a59185e5562e1a92253a454e08cd519 0 23 1121 1118 2021-06-04T10:37:04Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 5b4da8036d5a8bfd95d0c6dc6577a53dd9f25d18 1155 1121 2021-06-24T16:31:44Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Apache configuration snippet is now compatible with Apache 2.4 === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool add2012df9a3f4e319d83d657172634c689eafaa 1156 1155 2021-06-24T16:43:57Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool f7f7f1f61f12c83381a5768b1011b2ec708cf424 1157 1156 2021-07-16T11:42:43Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 0045aad7c5a8b3843c38bcf87620bf6e0b1046e1 1158 1157 2021-07-19T11:39:07Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 27e14151b9c7bfcffe3d35a1d17f475ce374f691 1159 1158 2021-07-20T11:05:24Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact''. * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 1a5e2f9c24fe0979b4c3332981a7b01275ba9253 1160 1159 2021-07-20T11:12:51Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 1d23002869d945d2f5ca7ec678148e13fcba9218 1161 1160 2021-09-16T15:18:26Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) ** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 7bb6284b26e1e02bb612da2a407bb7ee7b9340cc 1162 1161 2021-09-24T07:48:37Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Version 3.8.0 [2021-09-24] === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) ** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 8a8c57c2be9fe5051d3af228e52f191696052736 1163 1162 2021-09-24T07:48:49Z JoergHoletschek 3 /* Version 3.8.0 [2021-09-24] */ wikitext text/x-wiki == Current Version 3 == === Version 3.8.1 [2021-09-24] === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) ** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool f2564d7e05e99eb1177ecfac1d9c5b77fd1590ea 1164 1163 2021-12-07T13:53:39Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string) === Version 3.8.1 [2021-09-24] === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) ** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 28f6f9fabd7843937b10f08b75b9128f1ba5f4b6 1165 1164 2021-12-21T10:12:21Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === Fixes * Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string) * Check if file system supports Unicode; if not, convert archive file names to ASCII. === Version 3.8.1 [2021-09-24] === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) ** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 6185c055dfb4fae0090e57e8368c7d50a8ead3f4 1166 1165 2021-12-21T10:13:59Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === Fixes * Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string) * Check if file system supports Unicode; if not, convert archive file names to ASCII. * Fix for Ubuntu 18.04 - problems with Umlauts in archiving logs (Vienna Jacq server). === Version 3.8.1 [2021-09-24] === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) ** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool de7ac786ef759e00dca8a032911bf4b49a9bc27e 1167 1166 2021-12-21T16:10:01Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === Fixes * Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string) * Check if file system supports Unicode; if not, convert archive file names to ASCII. * Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduces in that version. === Version 3.8.1 [2021-09-24] === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) ** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 896581bcd5fc1b03a1843d0c4a496ca81f28ab80 1168 1167 2021-12-21T16:10:49Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === '''Important: BioCASe now requires Python version 3.7+''' * Fixes ** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string) ** Check if file system supports Unicode; if not, convert archive file names to ASCII. ** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduces in that version. === Version 3.8.1 [2021-09-24] === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) ** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool c5d386cd086c838452cab155f27ec7435bb3f5bb 1169 1168 2021-12-21T16:13:38Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === '''Important: BioCASe now requires Python version 3.7+''' * Fixes ** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string) ** Check if file system supports Unicode; if not, convert archive file names to ASCII. ** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore. === Version 3.8.1 [2021-09-24] === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) ** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 84971800159eaecf0f6e98d2bab7bea72e654b67 1170 1169 2021-12-22T09:51:13Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === '''Important: BioCASe now requires Python version 3.7+''' * Fixes ** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string) ** Check if file system supports Unicode; if not, convert archive file names to ASCII. ** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore. * Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed. === Version 3.8.1 [2021-09-24] === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) ** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 8a3e0586299ad4b8302ad047b8c046316369fad6 0 52 1122 1120 2021-06-07T11:06:46Z JoergHoletschek 3 /* Installation */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/ipt) you can create a DarwinCore Archive (see http://rs.tdwg.org/dwc/terms/guides/text/index.htm for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===After installing BioCASe, I get a "500: Internal Server Error"=== # Make sure you've [[Installation#Running_the_Setup_Script | run the setup script]] after downloading or checking out BioCASe with the right Python version. To check this, open the file ''www/index.cgi'' of your installation in a text editor and check the first line: This is the Python executable used by Python. # Empty the folder ''cache'' of your installation - please remove all folders and files you find here. # If the problem persists, please contact us! ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I'm using Internet Information Server, and after creating the Virtual Directory for BioCASe, I get a “502.2 - Bad Gateway. The specified CGI application misbehaved by not returning a complete set of HTTP headers” error.=== This happens when you didn't correctly associate .cgi and .py files with Python. Make sure you've linked both files with Python and that the path contains the placeholders ''%s %s'' (see [[Installation#Internet_Information_Server|Linking IIS]]). ===I cannot find the installation package of the database connectivity module for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later of psycopg2. ===I'm trying to use BioCASe with Postgres; for this, I tried to install psycopg on Windows. But it doesn't work - either the install crashes, or it finishes, but the test libs page still reports psycopg2 as not installed. In the debug log I can find the error message "ImportError: DLL load failed (...)". What's wrong?=== We suspect this is caused by a bug in recent psycopg2 releases (find more [http://psycopg.lighthouseapp.com/projects/62710/tickets/20 here]). On our test installations, we managed to get round this issue by installing pycopg2 with administration rights. Do do this, right-click the installer file and choose "Run as administrator". ===What's the difference between the DBMS packages odbc_tsql and odbc_tsql_2012 and between pymssql/pymssql_2012, respectively?=== The dbmods ending in "2012" use the new OFFSET clause that was introduced in MS SQL Server 2012. This will make paging faster and increase performance for larger datasets. They work only with SQL Server 2012 or later, and if you use SQL Server 2012, you should use them for best performance. Dbmods without a trailing "2012" work for both older versions and SQL Server 2012, but will be slower in paging large datasets. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== See [[Installation#Updating_an_existing_BioCASe_Installation|Updating an existing BioCASe Installation]]. ===I've updated my installation from a version prior to 2.6. Now, when I enter the DB structure, some of the table and column names are gone, the same happens in the mapping editor.=== Starting with version 2.6, BioCASe retrieves table and column names from the database for some database management systems (namely MySQL, SQL Server, Postgres, Foxpro, Access and Oracle). In older versions, you had to enter table/column names manually, and when you did the setup with such an early version, you probably didn't care for upper/lower cases. Since some DB management systems are case-sensitive, BioCASe can't ignore the differences and won't find these technically incorrect names in the lists retrieved from the database. The respective drop down boxes will therefore be empty. If that happens, just select the correct names from the drop down lists. ===When I try to create an archive, I get a "Permission denied" error. Why is that?=== For creating archives, you need to grant write permissions for the folders ''archive'' and ''www/downloads'' of your BioCASe installation to Python. You can check if permissions are set correctly in the test libs page (Start --> Utilities --> Test Libs). ===I'm trying to install BioCASe on Ubuntu (or another Linux). Even though I've run the setup script and added the snippet to the Apache configuration file, I see source code when I try out my installation instead of the pages described in the documentation=== Apparently CGI is not enabled on your machine. Google how to enable CGI for your distribution and follow the instructions you'll find. ===When trying to run BioCASe after installation, I get an internal server error (500). But the log folder of BioCASe is empty, nothing to find.=== This is probably because BioCASe - or the web server that runs BioCASe - can't write to this folder. Make sure that the web server (user ''www-data'' for Apache) has write permissions for this folder. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other only by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===I've mapped ABCD2. When testing the web service, I get a negative number for "RecordsDropped" in the response document. What does that mean?=== Same problem as above - it means you have an error in your mapping. Most probably you've mapped a non-repeatable ABCD concept to a table/column in your database that holds several entries per unit (occurrence) record. The BioCASe Provider Software will duplicate the respective unit records to be able to create valid ABCD documents - which is indicated by a negative number of "records dropped". Make sure the cardinality of your database relations match the structure (in respects of element repeatability) of ABCD to avoid this problem. ===I'm using pymssql to connect to a SQL Server database. Char(n) and Varchar(n) strings get truncated to 255 characters - what's wrong?=== This is known limitation of TDS protocol used by pymssql. Just change the data type for columns storing long string to TEXT or create a view that converts Char/Varchar columns to TEXT columns on the fly to work around this issue. ===I'm using pymssql to connect to a SQL Server database, BioCASe is running on a Linux distribution. When I do a Search or Scan request, I don't get any results (0 hits), even though there are definitely records in the database.=== That problem occurs on old versions of pymssql (1.0.2 or earlier) on Linux distributions. Update to version 2.0 or later to solve this. If you still get empty result sets, these posts might be helpful: * http://open-up.cybertaxonomy.africamuseum.be/forum_topic/important_if_you_use_biocase_bps_linux_platform_connect_ms_sql_server * http://code.google.com/p/pymssql/issues/detail?id=76. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===I am using BioCASe on a Windows Server. For some queries with the Local QueryForm, I get an error page that is obviously produced by the web server and not BioCASe. It reads "Error Code: 500 Internal Server Error. The request was rejected by the HTTP filter. Contact the server administrator. (12217)"=== This happens when BioCASe is run behind a Microsoft ISA Server/Microsoft Forefront Threat Management Gateway. To solve this problem, follow the instructions given in this article: http://support.microsoft.com/kb/837865. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. ===I'm using BioCASe in a Vagrant Virtual Box on MacOS with Synced Folders. When using the Manual Query Form or the Archiving page, I get an "ImportError: No module named wrapper.errorclasses."=== We don't know exactly what causes this, but apparently it happens when BioCASe is run in a Vagrant Virtual Box on MacOS and Vagrant's synced folder are used. We do have a workaround for this, but it is not easy. Please [[Special:Contact|contact us]] for assistance on this. ===I'm using BioCASe on a Windows server and odbc_tsql to connect to an MS SQL Server. Even though I do get results for a SCAN request, I cannot do a SEARCH (invalid document received).=== Please check your encoding settings in the DB connection configuration. If it is set to UTF_8, try Latin_1 instead. == Trouble Shooting == ===I’ve forgotten the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. 40c8d0a514e4d6664c320d293a0018437118e786 0 6 1123 971 2021-06-07T11:12:38Z JoergHoletschek 3 /* Installing Apache/IIS and Python */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|650px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, since you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 3 to run, it doesn't work with Python 2, which ended its life cycle in 2020. On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.) If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ac0484c87e696f8302535f5da148e0e05336242f 1125 1123 2021-06-07T12:30:19Z JoergHoletschek 3 /* Regular BioCASe Installation */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|650px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 3 to run, it doesn't work with Python 2, which ended its life cycle in 2020. On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.) If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_2.6.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. e75bdb251407db3842e6d275179ef9d55369890a 1126 1125 2021-06-07T12:32:06Z JoergHoletschek 3 /* Downloading from the Subversion Repository */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|650px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 3 to run, it doesn't work with Python 2, which ended its life cycle in 2020. On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.) If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_3.8.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|180px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|480px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. da1994fd075d04f5b1d67ad492bbf7c87aad6e20 1130 1126 2021-06-07T12:38:19Z JoergHoletschek 3 /* Downloading from the Subversion Repository */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|650px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 3 to run, it doesn't work with Python 2, which ended its life cycle in 2020. On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.) If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_3.8.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|250px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|650px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|480px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|480px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. f5befc2227f51127fcd1077170984b9251e89994 1132 1130 2021-06-07T12:39:38Z JoergHoletschek 3 /* Running the Setup Script */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|650px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 3 to run, it doesn't work with Python 2, which ended its life cycle in 2020. On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.) If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_3.8.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|250px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|650px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|550px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|550px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|480px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. 16ceff96ac12d3cfb3d5622dfddc23f1d907b037 1135 1132 2021-06-07T12:41:49Z JoergHoletschek 3 /* Apache */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|650px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 3 to run, it doesn't work with Python 2, which ended its life cycle in 2020. On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.) If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_3.8.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|250px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|650px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|550px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|550px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|550px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 * http://www.ecp.cc/pyiis.html ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. 1a7ed55395eb4d3e1433961bbc05c11508b8051e 1136 1135 2021-06-07T12:42:41Z JoergHoletschek 3 /* Internet Information Server */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|650px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 3 to run, it doesn't work with Python 2, which ended its life cycle in 2020. On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.) If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_3.8.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|250px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|650px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|550px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|550px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|550px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|500px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. 4591064d8f76299c4eda81d20b9119afc2c23ace 1138 1136 2021-06-07T12:43:27Z JoergHoletschek 3 /* Testing the Link */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|650px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 3 to run, it doesn't work with Python 2, which ended its life cycle in 2020. On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.) If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_3.8.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|250px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|650px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|550px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|550px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|550px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|550px]] === Installing Additional Python Packages/Java === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. de78a6dca22dd95d376a33288e979581436cd5b5 1139 1138 2021-06-07T12:44:06Z JoergHoletschek 3 /* Installing Additional Python Packages/Java */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|650px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 3 to run, it doesn't work with Python 2, which ended its life cycle in 2020. On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.) If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_3.8.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|250px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|650px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|550px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|550px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|550px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|550px]] === Installing Additional Python Packages/Java/Graphviz === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, it should look similar to this: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. cc5b12014dcf7fe617aa28168ec0b4f047eeedff 1140 1139 2021-06-07T13:02:10Z JoergHoletschek 3 /* Installing Additional Python Packages/Java/Graphviz */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|650px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 3 to run, it doesn't work with Python 2, which ended its life cycle in 2020. On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.) If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_3.8.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|250px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|650px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|550px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|550px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|550px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|550px]] === Installing Additional Python Packages/Java/Graphviz === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, only the BioCASe libraries should be installed: [[image:instLibrariesVanilla.png|450px]] The BioCASe libraries are already included in the archive file you’ve downloaded (or the Subversion repository you checked out from). The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. fec520b99bba363a04a1a3c4de78db6ec1220267 1142 1140 2021-06-07T13:04:27Z JoergHoletschek 3 /* Installing Additional Python Packages/Java/Graphviz */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|650px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 3 to run, it doesn't work with Python 2, which ended its life cycle in 2020. On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.) If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_3.8.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|250px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|650px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|550px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|550px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|550px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|550px]] === Installing Additional Python Packages/Java/Graphviz === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, only the BioCASe libraries should be installed, since they're part of the archive file you’ve downloaded (or the Subversion repository you checked out from): [[image:instLibrariesVanilla.png|450px]] The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ced02892e31520c3f1471249a7f69550b0a289eb 1143 1142 2021-06-07T13:05:10Z JoergHoletschek 3 /* Installing Additional Python Packages/Java/Graphviz */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|650px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 3 to run, it doesn't work with Python 2, which ended its life cycle in 2020. On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.) If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_3.8.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|250px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|650px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|550px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|550px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|550px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|550px]] === Installing Additional Python Packages/Java/Graphviz === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, only the BioCASe libraries should be installed, since they're part of the archive file you’ve downloaded (or the Subversion repository you checked out from): [[image:instLibrariesVanilla.png|550px]] The 4SuiteXML library is required only for the Local Query Tool – if you don’t plan to use it, you can ignore it. Otherwise click on the Download link, find the installer appropriate for your operating system and install it. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. 141f2e706b32f652fc68da124f639746a0b48c81 1144 1143 2021-06-07T13:09:44Z JoergHoletschek 3 /* Installing Additional Python Packages/Java/Graphviz */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|650px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 3 to run, it doesn't work with Python 2, which ended its life cycle in 2020. On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.) If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_3.8.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|250px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|650px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|550px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|550px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|550px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|550px]] === Installing Additional Python Packages/Java/Graphviz === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, only the BioCASe libraries should be installed, since they're part of the archive file you’ve downloaded (or the Subversion repository you checked out from): [[image:instLibrariesVanilla.png|550px]] If you plan to use the Local Query Tool (a small portal for querying and browsing your BioCASe webservice), you need to install the lxml library. You can do this easily by running the Python Package Installer (PIP): ''pip install lxml''. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, for example, click on the download link for the MySQL module, find the correct installer for your Python version and operating system and install. Once the installation is done, after reloading the libraries test page in the browser, the according list entry should turn from an orange ‘not installed’ into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. cd25e5b0498644c4213d6b062ace201e87e96fb7 1145 1144 2021-06-07T13:15:25Z JoergHoletschek 3 /* Installing Additional Python Packages/Java/Graphviz */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|650px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 3 to run, it doesn't work with Python 2, which ended its life cycle in 2020. On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.) If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_3.8.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|250px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|650px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|550px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|550px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|550px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|550px]] === Installing Additional Python Packages/Java/Graphviz === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, only the BioCASe libraries should be installed, since they're part of the archive file you’ve downloaded (or the Subversion repository you checked out from): [[image:instLibrariesVanilla.png|550px]] If you plan to use the Local Query Tool (a small portal for querying and browsing your BioCASe webservice), you need to install the lxml library. You can do this easily by running the Python Package Installer (PIP): ''pip install lxml''. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, you'll need the MySQLdb library; the table shows you the command to run for installing it using pip: ''pip install mysqlclient''. Once pip has installed the package, reload the libraries test page. The according list entry should turn from an orange ''not installed'' into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link, find the correct installer, and… you know the story. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). If it is installed on the machine, but the version is not shown as in the screenshot above, please specify the full path to the Java executable. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. 698d757cd116af07c06a749a48cf91e9b40b135e 1146 1145 2021-06-07T13:20:32Z JoergHoletschek 3 /* Installing Additional Python Packages/Java/Graphviz */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|650px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 3 to run, it doesn't work with Python 2, which ended its life cycle in 2020. On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.) If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_3.8.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|250px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|650px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|550px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|550px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|550px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|550px]] === Installing Additional Python Packages/Java/Graphviz === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, only the BioCASe libraries should be installed, since they're part of the archive file you’ve downloaded (or the Subversion repository you checked out from): [[image:instLibrariesVanilla.png|550px]] If you plan to use the Local Query Tool (a small portal for querying and browsing your BioCASe webservice), you need to install the lxml library. You can do this easily by running the Python Package Installer (PIP): ''pip install lxml''. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, you'll need the MySQLdb library; the table shows you the command to run for installing it using pip: ''pip install mysqlclient''. Once pip has installed the package, reload the libraries test page. The according list entry should turn from an orange ''not installed'' into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link and install the package using an installer file or a package manager on Linux. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). BioCASe requires Java 1.8 or later for creating DarwinCore archives, so please check the version listed on the Testlibs page if it's already installed on the server. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. f3588fba7e19ceac6df10b039f1a47954760d411 1148 1146 2021-06-07T13:24:45Z JoergHoletschek 3 /* Changing the Default Password */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|650px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 3 to run, it doesn't work with Python 2, which ended its life cycle in 2020. On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.) If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_3.8.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|250px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|650px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|550px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|550px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|550px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|550px]] === Installing Additional Python Packages/Java/Graphviz === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, only the BioCASe libraries should be installed, since they're part of the archive file you’ve downloaded (or the Subversion repository you checked out from): [[image:instLibrariesVanilla.png|550px]] If you plan to use the Local Query Tool (a small portal for querying and browsing your BioCASe webservice), you need to install the lxml library. You can do this easily by running the Python Package Installer (PIP): ''pip install lxml''. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, you'll need the MySQLdb library; the table shows you the command to run for installing it using pip: ''pip install mysqlclient''. Once pip has installed the package, reload the libraries test page. The according list entry should turn from an orange ''not installed'' into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link and install the package using an installer file or a package manager on Linux. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). BioCASe requires Java 1.8 or later for creating DarwinCore archives, so please check the version listed on the Testlibs page if it's already installed on the server. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Path to Graphviz executable</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:/Program Files/Graphviz 2.44.1/bin/''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for you DBMS, 4SuiteXML (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz Dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. Let’s assume you want to use BioCASe in conjunction with a MySQL and an Access database (which will be connected to through ODBC) and you don’t care for the Query Tool, the libs page should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. 52ef7b2e742c9a9f2fd07143b7c38e4d73a711e3 1149 1148 2021-06-07T13:32:27Z JoergHoletschek 3 /* Checking Your Installation */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|650px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 3 to run, it doesn't work with Python 2, which ended its life cycle in 2020. On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.) If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_3.8.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|250px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|650px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|550px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|550px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|550px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|550px]] === Installing Additional Python Packages/Java/Graphviz === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, only the BioCASe libraries should be installed, since they're part of the archive file you’ve downloaded (or the Subversion repository you checked out from): [[image:instLibrariesVanilla.png|550px]] If you plan to use the Local Query Tool (a small portal for querying and browsing your BioCASe webservice), you need to install the lxml library. You can do this easily by running the Python Package Installer (PIP): ''pip install lxml''. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, you'll need the MySQLdb library; the table shows you the command to run for installing it using pip: ''pip install mysqlclient''. Once pip has installed the package, reload the libraries test page. The according list entry should turn from an orange ''not installed'' into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link and install the package using an installer file or a package manager on Linux. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). BioCASe requires Java 1.8 or later for creating DarwinCore archives, so please check the version listed on the Testlibs page if it's already installed on the server. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Path to Graphviz executable</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:/Program Files/Graphviz 2.44.1/bin/''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for your DBMS, lxml (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. If you want to connect BioCASe to a variety of different database management systems, create DarwinCore archives and want to use the QueryTool, the libraries test pages should look like this: [[image:instLibrariesDone.png|450px]] == Updating an existing BioCASe Installation == === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. eba9f0d625588d9eec378c799d976adb9c5d0b28 1151 1149 2021-06-07T13:33:07Z JoergHoletschek 3 /* Checking Your Installation */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|650px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 3 to run, it doesn't work with Python 2, which ended its life cycle in 2020. On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.) If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_3.8.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|250px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|650px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|550px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|550px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|550px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|550px]] === Installing Additional Python Packages/Java/Graphviz === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, only the BioCASe libraries should be installed, since they're part of the archive file you’ve downloaded (or the Subversion repository you checked out from): [[image:instLibrariesVanilla.png|550px]] If you plan to use the Local Query Tool (a small portal for querying and browsing your BioCASe webservice), you need to install the lxml library. You can do this easily by running the Python Package Installer (PIP): ''pip install lxml''. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, you'll need the MySQLdb library; the table shows you the command to run for installing it using pip: ''pip install mysqlclient''. Once pip has installed the package, reload the libraries test page. The according list entry should turn from an orange ''not installed'' into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link and install the package using an installer file or a package manager on Linux. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). BioCASe requires Java 1.8 or later for creating DarwinCore archives, so please check the version listed on the Testlibs page if it's already installed on the server. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Path to Graphviz executable</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:/Program Files/Graphviz 2.44.1/bin/''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for your DBMS, lxml (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. If you want to connect BioCASe to a variety of different database management systems, create DarwinCore archives and want to use the QueryTool, the libraries test pages should look like this: [[image:instLibrariesDone.png|550px]] == Updating an existing BioCASe Installation == === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. 989ef403176e5ee026c80c0c817dffbd4bcbf2b3 1152 1151 2021-06-07T13:33:21Z JoergHoletschek 3 /* Checking Your Installation */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|650px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 3 to run, it doesn't work with Python 2, which ended its life cycle in 2020. On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.) If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_3.8.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|250px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|650px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|550px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|550px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|550px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|550px]] === Installing Additional Python Packages/Java/Graphviz === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, only the BioCASe libraries should be installed, since they're part of the archive file you’ve downloaded (or the Subversion repository you checked out from): [[image:instLibrariesVanilla.png|550px]] If you plan to use the Local Query Tool (a small portal for querying and browsing your BioCASe webservice), you need to install the lxml library. You can do this easily by running the Python Package Installer (PIP): ''pip install lxml''. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, you'll need the MySQLdb library; the table shows you the command to run for installing it using pip: ''pip install mysqlclient''. Once pip has installed the package, reload the libraries test page. The according list entry should turn from an orange ''not installed'' into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link and install the package using an installer file or a package manager on Linux. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). BioCASe requires Java 1.8 or later for creating DarwinCore archives, so please check the version listed on the Testlibs page if it's already installed on the server. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Path to Graphviz executable</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:/Program Files/Graphviz 2.44.1/bin/''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for your DBMS, lxml (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. If you want to connect BioCASe to a variety of different database management systems, create DarwinCore archives and want to use the QueryTool, the libraries test pages should look like this: [[image:instLibrariesDone.png|650px]] == Updating an existing BioCASe Installation == === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. 13c1ede499609a58c0c108ed8454099667ad90ab 1153 1152 2021-06-07T13:33:49Z JoergHoletschek 3 /* Checking Your Installation */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|650px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 3 to run, it doesn't work with Python 2, which ended its life cycle in 2020. On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.) If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_3.8.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|250px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|650px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|550px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|550px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|550px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|550px]] === Installing Additional Python Packages/Java/Graphviz === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, only the BioCASe libraries should be installed, since they're part of the archive file you’ve downloaded (or the Subversion repository you checked out from): [[image:instLibrariesVanilla.png|550px]] If you plan to use the Local Query Tool (a small portal for querying and browsing your BioCASe webservice), you need to install the lxml library. You can do this easily by running the Python Package Installer (PIP): ''pip install lxml''. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, you'll need the MySQLdb library; the table shows you the command to run for installing it using pip: ''pip install mysqlclient''. Once pip has installed the package, reload the libraries test page. The according list entry should turn from an orange ''not installed'' into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link and install the package using an installer file or a package manager on Linux. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). BioCASe requires Java 1.8 or later for creating DarwinCore archives, so please check the version listed on the Testlibs page if it's already installed on the server. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Path to Graphviz executable</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:/Program Files/Graphviz 2.44.1/bin/''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for your DBMS, lxml (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. If you want to connect BioCASe to a variety of different database management systems, create DarwinCore archives and use the QueryTool, the libraries test pages should look like this: [[image:instLibrariesDone.png|650px]] == Updating an existing BioCASe Installation == === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. 4d6c6b2fa88a33679f254cf1bf16749facadc70e 1154 1153 2021-06-07T13:42:58Z JoergHoletschek 3 /* Downloading from the Subversion Repository */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|650px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 3 to run, it doesn't work with Python 2, which ended its life cycle in 2020. On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.) If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_3.8.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|300px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|650px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|550px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|550px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|550px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|550px]] === Installing Additional Python Packages/Java/Graphviz === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, only the BioCASe libraries should be installed, since they're part of the archive file you’ve downloaded (or the Subversion repository you checked out from): [[image:instLibrariesVanilla.png|550px]] If you plan to use the Local Query Tool (a small portal for querying and browsing your BioCASe webservice), you need to install the lxml library. You can do this easily by running the Python Package Installer (PIP): ''pip install lxml''. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, you'll need the MySQLdb library; the table shows you the command to run for installing it using pip: ''pip install mysqlclient''. Once pip has installed the package, reload the libraries test page. The according list entry should turn from an orange ''not installed'' into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link and install the package using an installer file or a package manager on Linux. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). BioCASe requires Java 1.8 or later for creating DarwinCore archives, so please check the version listed on the Testlibs page if it's already installed on the server. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Path to Graphviz executable</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:/Program Files/Graphviz 2.44.1/bin/''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for your DBMS, lxml (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. If you want to connect BioCASe to a variety of different database management systems, create DarwinCore archives and use the QueryTool, the libraries test pages should look like this: [[image:instLibrariesDone.png|650px]] == Updating an existing BioCASe Installation == === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. 5ac50825fae507761eb367ee0943b318760d725e 6 114 1124 951 2021-06-07T12:28:44Z JoergHoletschek 3 JoergHoletschek uploaded a new version of [[File:DockerInstall.png]] wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 9 1127 18 2021-06-07T12:32:34Z JoergHoletschek 3 JoergHoletschek uploaded a new version of [[File:InstTortoiseDialog.png]] wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 8 1128 17 2021-06-07T12:34:59Z JoergHoletschek 3 JoergHoletschek uploaded a new version of [[File:InstTortoiseMenu.png]] wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 7 1129 16 2021-06-07T12:36:33Z JoergHoletschek 3 JoergHoletschek uploaded a new version of [[File:InstBiocaseFolder.png]] wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 10 1131 342 2021-06-07T12:38:55Z JoergHoletschek 3 JoergHoletschek uploaded a new version of [[File:InstSetupScriptStart.png]] wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 11 1133 22 2021-06-07T12:40:09Z JoergHoletschek 3 JoergHoletschek uploaded a new version of [[File:InstSetupScriptDone.png]] wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 12 1134 26 2021-06-07T12:40:59Z JoergHoletschek 3 JoergHoletschek uploaded a new version of [[File:InstHttpdConf.png]] wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 13 1137 27 2021-06-07T12:43:07Z JoergHoletschek 3 JoergHoletschek uploaded a new version of [[File:InstBiocaseStartpage.png]] wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 14 1141 588 2021-06-07T13:02:31Z JoergHoletschek 3 JoergHoletschek uploaded a new version of [[File:InstLibrariesVanilla.png]] wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 15 1147 32 2021-06-07T13:21:59Z JoergHoletschek 3 JoergHoletschek uploaded a new version of [[File:InstConfigTool.png]] wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 6 16 1150 592 2021-06-07T13:32:44Z JoergHoletschek 3 JoergHoletschek uploaded a new version of [[File:InstLibrariesDone.png]] wikitext text/x-wiki da39a3ee5e6b4b0d3255bfef95601890afd80709 0 23 1171 1170 2022-06-08T14:51:26Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Version 3.8.2 [2021-06-08]=== '''Important: BioCASe now requires Python version 3.7+''' * Fixes ** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string) ** Check if file system supports Unicode; if not, convert archive file names to ASCII. ** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore. * Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed. === Version 3.8.1 [2021-09-24] === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) ** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool de327cb8b58aca960805e55bbab3c83936f8c9b6 1172 1171 2022-06-08T14:51:37Z JoergHoletschek 3 /* Version 3.8.2 [2021-06-08] */ wikitext text/x-wiki == Current Version 3 == === Version 3.8.2 [2022-06-08]=== '''Important: BioCASe now requires Python version 3.7+''' * Fixes ** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string) ** Check if file system supports Unicode; if not, convert archive file names to ASCII. ** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore. * Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed. === Version 3.8.1 [2021-09-24] === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) ** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool a14c69dbe5acf5db156af163875d26e47a74939d 1174 1172 2023-01-13T09:52:48Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === 3.8.3 [2023-01-xx]=== * Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10) === Version 3.8.2 [2022-06-08]=== '''Important: BioCASe now requires Python version 3.7+''' * Fixes ** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string) ** Check if file system supports Unicode; if not, convert archive file names to ASCII. ** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore. * Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed. === Version 3.8.1 [2021-09-24] === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) ** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 90f61f6994750ac81bfd0aea4b078dde64e855c7 1175 1174 2023-01-13T12:12:54Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Version 3.8.3 [2023-01-13]=== * Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10) === Version 3.8.2 [2022-06-08]=== '''Important: BioCASe now requires Python version 3.7+''' * Fixes ** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string) ** Check if file system supports Unicode; if not, convert archive file names to ASCII. ** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore. * Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed. === Version 3.8.1 [2021-09-24] === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) ** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 3dbe382bc0f2ea957e31ea2ec71dac8fbf270841 1176 1175 2023-04-18T10:19:01Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Fix for login page of configtool (didn't used the webroot from configuration) === Version 3.8.3 [2023-01-13]=== * Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10) === Version 3.8.2 [2022-06-08]=== '''Important: BioCASe now requires Python version 3.7+''' * Fixes ** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string) ** Check if file system supports Unicode; if not, convert archive file names to ASCII. ** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore. * Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed. === Version 3.8.1 [2021-09-24] === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) ** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 2c8b3f61e8f6d6db18e228696e1cce32ed40cd04 1177 1176 2023-08-18T11:59:51Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Fix for login page of configtool (didn't used the webroot from configuration) * Use SQL statements with parameters for most DB mods === Version 3.8.3 [2023-01-13]=== * Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10) === Version 3.8.2 [2022-06-08]=== '''Important: BioCASe now requires Python version 3.7+''' * Fixes ** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string) ** Check if file system supports Unicode; if not, convert archive file names to ASCII. ** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore. * Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed. === Version 3.8.1 [2021-09-24] === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) ** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 24ca2b444c7dcbf2eef79e9449d4fce0b7597615 1178 1177 2023-08-18T12:03:21Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Version 3.8.4 [2023-08-18] === * Fix for login page of configtool (didn't used the webroot from configuration) * Use SQL statements with parameters for most DB mods === Version 3.8.3 [2023-01-13]=== * Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10) === Version 3.8.2 [2022-06-08]=== '''Important: BioCASe now requires Python version 3.7+''' * Fixes ** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string) ** Check if file system supports Unicode; if not, convert archive file names to ASCII. ** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore. * Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed. === Version 3.8.1 [2021-09-24] === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) ** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 716398cfd24632659f5c11bad09d3f356631e66b 1179 1178 2023-09-29T13:50:29Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Bugfix for empty webroots (like biocase.xxxx.de) === Version 3.8.4 [2023-08-18] === * Fix for login page of configtool (didn't used the webroot from configuration) * Use SQL statements with parameters for most DB mods === Version 3.8.3 [2023-01-13]=== * Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10) === Version 3.8.2 [2022-06-08]=== '''Important: BioCASe now requires Python version 3.7+''' * Fixes ** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string) ** Check if file system supports Unicode; if not, convert archive file names to ASCII. ** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore. * Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed. === Version 3.8.1 [2021-09-24] === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) ** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 1e32becb0bcef8a03838f30b49ed2f7cc99ea54c 1180 1179 2023-10-02T10:18:57Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'') * Fix for CMF upgrade function === Version 3.8.4 [2023-08-18] === * Fix for login page of configtool (didn't used the webroot from configuration) * Use SQL statements with parameters for most DB mods === Version 3.8.3 [2023-01-13]=== * Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10) === Version 3.8.2 [2022-06-08]=== '''Important: BioCASe now requires Python version 3.7+''' * Fixes ** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string) ** Check if file system supports Unicode; if not, convert archive file names to ASCII. ** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore. * Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed. === Version 3.8.1 [2021-09-24] === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) ** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 9c7ca59249f72b8e2ee725a05bac5d1a1a9b6556 1181 1180 2023-11-30T11:14:04Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Version 3.8.5 [2023-11-30] === * Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'') * Fix for CMF upgrade function === Version 3.8.4 [2023-08-18] === * Fix for login page of configtool (didn't used the webroot from configuration) * Use SQL statements with parameters for most DB mods === Version 3.8.3 [2023-01-13]=== * Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10) === Version 3.8.2 [2022-06-08]=== '''Important: BioCASe now requires Python version 3.7+''' * Fixes ** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string) ** Check if file system supports Unicode; if not, convert archive file names to ASCII. ** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore. * Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed. === Version 3.8.1 [2021-09-24] === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) ** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 6ccd84f2d312ab52ea2fd4eac8e071bcc07b8da6 1182 1181 2024-04-15T13:50:29Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Timeout added for the request to retrieve latest version from sbuversion repository === Version 3.8.5 [2023-11-30] === * Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'') * Fix for CMF upgrade function === Version 3.8.4 [2023-08-18] === * Fix for login page of configtool (didn't used the webroot from configuration) * Use SQL statements with parameters for most DB mods === Version 3.8.3 [2023-01-13]=== * Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10) === Version 3.8.2 [2022-06-08]=== '''Important: BioCASe now requires Python version 3.7+''' * Fixes ** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string) ** Check if file system supports Unicode; if not, convert archive file names to ASCII. ** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore. * Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed. === Version 3.8.1 [2021-09-24] === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) ** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool ca8c08834c5a7fda3c2556e21fbf988e97877df4 1184 1182 2024-07-09T07:46:28Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Timeout added for the request to retrieve latest version from subversion repository * Archiving page: List schemas in dropdown box in descending order, so that ABCD 2.1 shows before 2.06 before 1.2 === Version 3.8.5 [2023-11-30] === * Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'') * Fix for CMF upgrade function === Version 3.8.4 [2023-08-18] === * Fix for login page of configtool (didn't used the webroot from configuration) * Use SQL statements with parameters for most DB mods === Version 3.8.3 [2023-01-13]=== * Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10) === Version 3.8.2 [2022-06-08]=== '''Important: BioCASe now requires Python version 3.7+''' * Fixes ** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string) ** Check if file system supports Unicode; if not, convert archive file names to ASCII. ** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore. * Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed. === Version 3.8.1 [2021-09-24] === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) ** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 3c68cf34e95be61a712cdf46e31888de2163e585 1186 1184 2024-11-04T11:51:26Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Timeout added for the request to retrieve latest version from subversion repository * Archiving page: List schemas in dropdown box in descending order, so that ABCD 2.1 shows before 2.06 before 1.2 * QueryTool: Changed default schema from ABCD 2.1 to ABCD 2.06, since this is used more commonly === Version 3.8.5 [2023-11-30] === * Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'') * Fix for CMF upgrade function === Version 3.8.4 [2023-08-18] === * Fix for login page of configtool (didn't used the webroot from configuration) * Use SQL statements with parameters for most DB mods === Version 3.8.3 [2023-01-13]=== * Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10) === Version 3.8.2 [2022-06-08]=== '''Important: BioCASe now requires Python version 3.7+''' * Fixes ** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string) ** Check if file system supports Unicode; if not, convert archive file names to ASCII. ** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore. * Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed. === Version 3.8.1 [2021-09-24] === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) ** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool b6f7597556b09744ddc46ceae0fe88648bdfe8db 1187 1186 2024-11-14T10:40:08Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Timeout added for the request to retrieve latest version from subversion repository * Archiving page: List schemas in dropdown box in descending order, so that ABCD 2.1 shows before 2.06 before 1.2 * QueryTool: Changed default schema from ABCD 2.1 to ABCD 2.06, since this is used more commonly * Fix for DwC archives: occurrenceStatus and occurrencID had been swapped in descriptor file meta.xml === Version 3.8.5 [2023-11-30] === * Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'') * Fix for CMF upgrade function === Version 3.8.4 [2023-08-18] === * Fix for login page of configtool (didn't used the webroot from configuration) * Use SQL statements with parameters for most DB mods === Version 3.8.3 [2023-01-13]=== * Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10) === Version 3.8.2 [2022-06-08]=== '''Important: BioCASe now requires Python version 3.7+''' * Fixes ** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string) ** Check if file system supports Unicode; if not, convert archive file names to ASCII. ** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore. * Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed. === Version 3.8.1 [2021-09-24] === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) ** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool efd1155dee7b85ed98d98c38b9e989d4c37225d1 1188 1187 2024-11-14T15:06:43Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Timeout added for the request to retrieve latest version from subversion repository * Archiving page: List schemas in dropdown box in descending order, so that ABCD 2.1 shows before 2.06 before 1.2 * QueryTool: Changed default schema from ABCD 2.1 to ABCD 2.06, since this is used more commonly * Fix for DwC archives: occurrenceStatus and occurrencID had been swapped in descriptor file meta.xml * DwC archive transformation: BasisOfRecord="AbsenceObservation" will be translated into "HumanObservation", since AbsenceObservation is not accepted by GBIF. === Version 3.8.5 [2023-11-30] === * Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'') * Fix for CMF upgrade function === Version 3.8.4 [2023-08-18] === * Fix for login page of configtool (didn't used the webroot from configuration) * Use SQL statements with parameters for most DB mods === Version 3.8.3 [2023-01-13]=== * Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10) === Version 3.8.2 [2022-06-08]=== '''Important: BioCASe now requires Python version 3.7+''' * Fixes ** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string) ** Check if file system supports Unicode; if not, convert archive file names to ASCII. ** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore. * Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed. === Version 3.8.1 [2021-09-24] === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) ** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 5b09b41263fa8614a5af21144418c5db57234671 1189 1188 2024-12-12T08:51:32Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Version Version 3.8.5 [2023-11-30] [2024-12-12] === * Timeout added for the request to retrieve latest version from subversion repository * Archiving page: List schemas in dropdown box in descending order, so that ABCD 2.1 shows before 2.06 before 1.2 * QueryTool: Changed default schema from ABCD 2.1 to ABCD 2.06, since this is used more commonly * Fix for DwC archives: occurrenceStatus and occurrencID had been swapped in descriptor file meta.xml * DwC archive transformation: BasisOfRecord="AbsenceObservation" will be translated into "HumanObservation", since AbsenceObservation is not accepted by GBIF. === Version 3.8.5 [2023-11-30] === * Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'') * Fix for CMF upgrade function === Version 3.8.4 [2023-08-18] === * Fix for login page of configtool (didn't used the webroot from configuration) * Use SQL statements with parameters for most DB mods === Version 3.8.3 [2023-01-13]=== * Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10) === Version 3.8.2 [2022-06-08]=== '''Important: BioCASe now requires Python version 3.7+''' * Fixes ** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string) ** Check if file system supports Unicode; if not, convert archive file names to ASCII. ** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore. * Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed. === Version 3.8.1 [2021-09-24] === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) ** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool eb086bef75b46f8e6f946e02556c6ef4e26320b8 1190 1189 2024-12-12T08:51:52Z JoergHoletschek 3 /* Version Version 3.8.5 [2023-11-30] [2024-12-12] */ wikitext text/x-wiki == Current Version 3 == === Version Version 3.8.6 [2024-12-12] === * Timeout added for the request to retrieve latest version from subversion repository * Archiving page: List schemas in dropdown box in descending order, so that ABCD 2.1 shows before 2.06 before 1.2 * QueryTool: Changed default schema from ABCD 2.1 to ABCD 2.06, since this is used more commonly * Fix for DwC archives: occurrenceStatus and occurrencID had been swapped in descriptor file meta.xml * DwC archive transformation: BasisOfRecord="AbsenceObservation" will be translated into "HumanObservation", since AbsenceObservation is not accepted by GBIF. === Version 3.8.5 [2023-11-30] === * Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'') * Fix for CMF upgrade function === Version 3.8.4 [2023-08-18] === * Fix for login page of configtool (didn't used the webroot from configuration) * Use SQL statements with parameters for most DB mods === Version 3.8.3 [2023-01-13]=== * Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10) === Version 3.8.2 [2022-06-08]=== '''Important: BioCASe now requires Python version 3.7+''' * Fixes ** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string) ** Check if file system supports Unicode; if not, convert archive file names to ASCII. ** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore. * Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed. === Version 3.8.1 [2021-09-24] === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) ** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool ea4b78f6fc9e852aadcbcf48c0a2f1b3eddad09c 1191 1190 2024-12-19T14:50:44Z JoergHoletschek 3 /* Version Version 3.8.6 [2024-12-12] */ wikitext text/x-wiki == Current Version 3 == === Version 3.8.6 [2024-12-12] === * Timeout added for the request to retrieve latest version from subversion repository * Archiving page: List schemas in dropdown box in descending order, so that ABCD 2.1 shows before 2.06 before 1.2 * QueryTool: Changed default schema from ABCD 2.1 to ABCD 2.06, since this is used more commonly * Fix for DwC archives: occurrenceStatus and occurrencID had been swapped in descriptor file meta.xml * DwC archive transformation: BasisOfRecord="AbsenceObservation" will be translated into "HumanObservation", since AbsenceObservation is not accepted by GBIF. === Version 3.8.5 [2023-11-30] === * Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'') * Fix for CMF upgrade function === Version 3.8.4 [2023-08-18] === * Fix for login page of configtool (didn't used the webroot from configuration) * Use SQL statements with parameters for most DB mods === Version 3.8.3 [2023-01-13]=== * Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10) === Version 3.8.2 [2022-06-08]=== '''Important: BioCASe now requires Python version 3.7+''' * Fixes ** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string) ** Check if file system supports Unicode; if not, convert archive file names to ASCII. ** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore. * Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed. === Version 3.8.1 [2021-09-24] === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) ** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 9dec8ade17019699680a8d7c28e22e1de4a8d992 1192 1191 2024-12-19T14:51:03Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Dockerfile adapted to latest Debian version and Python 3.13 === Version 3.8.6 [2024-12-12] === * Timeout added for the request to retrieve latest version from subversion repository * Archiving page: List schemas in dropdown box in descending order, so that ABCD 2.1 shows before 2.06 before 1.2 * QueryTool: Changed default schema from ABCD 2.1 to ABCD 2.06, since this is used more commonly * Fix for DwC archives: occurrenceStatus and occurrencID had been swapped in descriptor file meta.xml * DwC archive transformation: BasisOfRecord="AbsenceObservation" will be translated into "HumanObservation", since AbsenceObservation is not accepted by GBIF. === Version 3.8.5 [2023-11-30] === * Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'') * Fix for CMF upgrade function === Version 3.8.4 [2023-08-18] === * Fix for login page of configtool (didn't used the webroot from configuration) * Use SQL statements with parameters for most DB mods === Version 3.8.3 [2023-01-13]=== * Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10) === Version 3.8.2 [2022-06-08]=== '''Important: BioCASe now requires Python version 3.7+''' * Fixes ** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string) ** Check if file system supports Unicode; if not, convert archive file names to ASCII. ** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore. * Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed. === Version 3.8.1 [2021-09-24] === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) ** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 2b0e8774eace1363371dd664b01d21d70c250304 1193 1192 2025-01-21T09:11:38Z JoergHoletschek 3 /* Current Version 3 */ wikitext text/x-wiki == Current Version 3 == === Upcoming === === Version 3.8.6 [2024-12-12] === * Timeout added for the request to retrieve latest version from subversion repository * Archiving page: List schemas in dropdown box in descending order, so that ABCD 2.1 shows before 2.06 before 1.2 * QueryTool: Changed default schema from ABCD 2.1 to ABCD 2.06, since this is used more commonly * Fix for DwC archives: occurrenceStatus and occurrencID had been swapped in descriptor file meta.xml * DwC archive transformation: BasisOfRecord="AbsenceObservation" will be translated into "HumanObservation", since AbsenceObservation is not accepted by GBIF. * Dockerfile adapted to latest Debian version and Python 3.13 === Version 3.8.5 [2023-11-30] === * Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'') * Fix for CMF upgrade function === Version 3.8.4 [2023-08-18] === * Fix for login page of configtool (didn't used the webroot from configuration) * Use SQL statements with parameters for most DB mods === Version 3.8.3 [2023-01-13]=== * Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10) === Version 3.8.2 [2022-06-08]=== '''Important: BioCASe now requires Python version 3.7+''' * Fixes ** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string) ** Check if file system supports Unicode; if not, convert archive file names to ASCII. ** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore. * Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed. === Version 3.8.1 [2021-09-24] === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) ** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 672fffc8870ff9905cc8cffb6bbd5a66313f513e 1194 1193 2025-02-11T11:03:16Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Update hint now uses link to new Git repo instead of Subversion === Version 3.8.6 [2024-12-12] === * Timeout added for the request to retrieve latest version from subversion repository * Archiving page: List schemas in dropdown box in descending order, so that ABCD 2.1 shows before 2.06 before 1.2 * QueryTool: Changed default schema from ABCD 2.1 to ABCD 2.06, since this is used more commonly * Fix for DwC archives: occurrenceStatus and occurrencID had been swapped in descriptor file meta.xml * DwC archive transformation: BasisOfRecord="AbsenceObservation" will be translated into "HumanObservation", since AbsenceObservation is not accepted by GBIF. * Dockerfile adapted to latest Debian version and Python 3.13 === Version 3.8.5 [2023-11-30] === * Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'') * Fix for CMF upgrade function === Version 3.8.4 [2023-08-18] === * Fix for login page of configtool (didn't used the webroot from configuration) * Use SQL statements with parameters for most DB mods === Version 3.8.3 [2023-01-13]=== * Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10) === Version 3.8.2 [2022-06-08]=== '''Important: BioCASe now requires Python version 3.7+''' * Fixes ** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string) ** Check if file system supports Unicode; if not, convert archive file names to ASCII. ** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore. * Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed. === Version 3.8.1 [2021-09-24] === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) ** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool ad3e7bf603541b36bccbbe5952c9f81c28262722 1200 1194 2025-02-13T12:07:19Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Update hint now uses link to new Git repo instead of Subversion * Use alternative DB connector pymysql if importing MySQLdb fails (apparently on latest Ubuntu distributions). Thanks to Kjell-Åke Lundblad from Nordic Genetic Resource Center for this! === Version 3.8.6 [2024-12-12] === * Timeout added for the request to retrieve latest version from subversion repository * Archiving page: List schemas in dropdown box in descending order, so that ABCD 2.1 shows before 2.06 before 1.2 * QueryTool: Changed default schema from ABCD 2.1 to ABCD 2.06, since this is used more commonly * Fix for DwC archives: occurrenceStatus and occurrencID had been swapped in descriptor file meta.xml * DwC archive transformation: BasisOfRecord="AbsenceObservation" will be translated into "HumanObservation", since AbsenceObservation is not accepted by GBIF. * Dockerfile adapted to latest Debian version and Python 3.13 === Version 3.8.5 [2023-11-30] === * Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'') * Fix for CMF upgrade function === Version 3.8.4 [2023-08-18] === * Fix for login page of configtool (didn't used the webroot from configuration) * Use SQL statements with parameters for most DB mods === Version 3.8.3 [2023-01-13]=== * Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10) === Version 3.8.2 [2022-06-08]=== '''Important: BioCASe now requires Python version 3.7+''' * Fixes ** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string) ** Check if file system supports Unicode; if not, convert archive file names to ASCII. ** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore. * Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed. === Version 3.8.1 [2021-09-24] === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) ** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 7c16102daa5812c8920bfcc4e204bac6cefacfcc 1201 1200 2025-03-04T12:16:42Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Upcoming === * Code repo moved to git: https://git.bgbm.org/biocase/bps * Update hint now uses link to new Git repo instead of Subversion * Use alternative DB connector pymysql if importing MySQLdb fails (apparently on latest Ubuntu distributions). Thanks to Kjell-Åke Lundblad from Nordic Genetic Resource Center for this! * Miscellaneous optimisations in Docker image (multistage build & chained RUN statements for smaller image size, separate Apache config file, no root user) thanks to Petr Novotný from Prague university! === Version 3.8.6 [2024-12-12] === * Timeout added for the request to retrieve latest version from subversion repository * Archiving page: List schemas in dropdown box in descending order, so that ABCD 2.1 shows before 2.06 before 1.2 * QueryTool: Changed default schema from ABCD 2.1 to ABCD 2.06, since this is used more commonly * Fix for DwC archives: occurrenceStatus and occurrencID had been swapped in descriptor file meta.xml * DwC archive transformation: BasisOfRecord="AbsenceObservation" will be translated into "HumanObservation", since AbsenceObservation is not accepted by GBIF. * Dockerfile adapted to latest Debian version and Python 3.13 === Version 3.8.5 [2023-11-30] === * Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'') * Fix for CMF upgrade function === Version 3.8.4 [2023-08-18] === * Fix for login page of configtool (didn't used the webroot from configuration) * Use SQL statements with parameters for most DB mods === Version 3.8.3 [2023-01-13]=== * Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10) === Version 3.8.2 [2022-06-08]=== '''Important: BioCASe now requires Python version 3.7+''' * Fixes ** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string) ** Check if file system supports Unicode; if not, convert archive file names to ASCII. ** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore. * Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed. === Version 3.8.1 [2021-09-24] === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) ** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool a4bb0f2e8b6e08059d9f1a41fbc3e1f13d81c8f1 1202 1201 2025-03-04T14:03:28Z JoergHoletschek 3 /* Upcoming */ wikitext text/x-wiki == Current Version 3 == === Version 3.8.7 [2025-03-04] === * Code repo moved to git: https://git.bgbm.org/biocase/bps * Update hint now uses link to new Git repo instead of Subversion * Use alternative DB connector pymysql if importing MySQLdb fails (apparently on latest Ubuntu distributions). Thanks to Kjell-Åke Lundblad from Nordic Genetic Resource Center for this! * Miscellaneous optimisations in Docker image (multistage build & chained RUN statements for smaller image size, separate Apache config file, no root user) thanks to Petr Novotný from Prague university! === Version 3.8.6 [2024-12-12] === * Timeout added for the request to retrieve latest version from subversion repository * Archiving page: List schemas in dropdown box in descending order, so that ABCD 2.1 shows before 2.06 before 1.2 * QueryTool: Changed default schema from ABCD 2.1 to ABCD 2.06, since this is used more commonly * Fix for DwC archives: occurrenceStatus and occurrencID had been swapped in descriptor file meta.xml * DwC archive transformation: BasisOfRecord="AbsenceObservation" will be translated into "HumanObservation", since AbsenceObservation is not accepted by GBIF. * Dockerfile adapted to latest Debian version and Python 3.13 === Version 3.8.5 [2023-11-30] === * Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'') * Fix for CMF upgrade function === Version 3.8.4 [2023-08-18] === * Fix for login page of configtool (didn't used the webroot from configuration) * Use SQL statements with parameters for most DB mods === Version 3.8.3 [2023-01-13]=== * Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10) === Version 3.8.2 [2022-06-08]=== '''Important: BioCASe now requires Python version 3.7+''' * Fixes ** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string) ** Check if file system supports Unicode; if not, convert archive file names to ASCII. ** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore. * Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed. === Version 3.8.1 [2021-09-24] === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) ** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool fcce7d87123647033caf7a3116a20d5b4263e798 1203 1202 2025-03-05T10:23:25Z JoergHoletschek 3 /* Version 3.8.7 [2025-03-04] */ wikitext text/x-wiki == Current Version 3 == === Version 3.8.7 [Upcoming] === * Code repo moved to git: https://git.bgbm.org/biocase/bps * Update hint now uses link to new Git repo instead of Subversion * Use alternative DB connector pymysql if importing MySQLdb fails (apparently on latest Ubuntu distributions). Thanks to Kjell-Åke Lundblad from Nordic Genetic Resource Center for this! * Miscellaneous optimisations in Docker image (multistage build & chained RUN statements for smaller image size, separate Apache config file, no root user) thanks to Petr Novotný from Prague university! === Version 3.8.6 [2024-12-12] === * Timeout added for the request to retrieve latest version from subversion repository * Archiving page: List schemas in dropdown box in descending order, so that ABCD 2.1 shows before 2.06 before 1.2 * QueryTool: Changed default schema from ABCD 2.1 to ABCD 2.06, since this is used more commonly * Fix for DwC archives: occurrenceStatus and occurrencID had been swapped in descriptor file meta.xml * DwC archive transformation: BasisOfRecord="AbsenceObservation" will be translated into "HumanObservation", since AbsenceObservation is not accepted by GBIF. * Dockerfile adapted to latest Debian version and Python 3.13 === Version 3.8.5 [2023-11-30] === * Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'') * Fix for CMF upgrade function === Version 3.8.4 [2023-08-18] === * Fix for login page of configtool (didn't used the webroot from configuration) * Use SQL statements with parameters for most DB mods === Version 3.8.3 [2023-01-13]=== * Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10) === Version 3.8.2 [2022-06-08]=== '''Important: BioCASe now requires Python version 3.7+''' * Fixes ** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string) ** Check if file system supports Unicode; if not, convert archive file names to ASCII. ** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore. * Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed. === Version 3.8.1 [2021-09-24] === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) ** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 04b0eb4ea6ea1521165f285ae574ce5e9fd6ab43 1204 1203 2025-03-06T10:04:55Z JoergHoletschek 3 /* Version 3.8.7 [Upcoming] */ wikitext text/x-wiki == Current Version 3 == === Version 3.8.7 [2025-03-06] === * Code repo moved to git: https://git.bgbm.org/biocase/bps * Update hint now uses link to new Git repo instead of Subversion * Use alternative DB connector pymysql if importing MySQLdb fails (apparently on latest Ubuntu distributions). Thanks to Kjell-Åke Lundblad from Nordic Genetic Resource Center for this! * Miscellaneous optimisations in Docker image (multistage build & chained RUN statements for smaller image size, separate Apache config file, no root user) thanks to Petr Novotný from Prague university! === Version 3.8.6 [2024-12-12] === * Timeout added for the request to retrieve latest version from subversion repository * Archiving page: List schemas in dropdown box in descending order, so that ABCD 2.1 shows before 2.06 before 1.2 * QueryTool: Changed default schema from ABCD 2.1 to ABCD 2.06, since this is used more commonly * Fix for DwC archives: occurrenceStatus and occurrencID had been swapped in descriptor file meta.xml * DwC archive transformation: BasisOfRecord="AbsenceObservation" will be translated into "HumanObservation", since AbsenceObservation is not accepted by GBIF. * Dockerfile adapted to latest Debian version and Python 3.13 === Version 3.8.5 [2023-11-30] === * Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'') * Fix for CMF upgrade function === Version 3.8.4 [2023-08-18] === * Fix for login page of configtool (didn't used the webroot from configuration) * Use SQL statements with parameters for most DB mods === Version 3.8.3 [2023-01-13]=== * Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10) === Version 3.8.2 [2022-06-08]=== '''Important: BioCASe now requires Python version 3.7+''' * Fixes ** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string) ** Check if file system supports Unicode; if not, convert archive file names to ASCII. ** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore. * Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed. === Version 3.8.1 [2021-09-24] === * DarwinCore Archives ** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'') ** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact'' ** Invalid term ''occurrenceDetails'' removed * Smaller fixes ** Apache configuration snippet is now compatible with Apache 2.4 ** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0) ** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading) ** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems) === Version 3.8.0 [2021-06-04] === * BioCASe now runs on Python 3 - yeah! * Advantages ** Removal of external dependencies: consistent use of internal Python packages ** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases) ** Better handling of invalid database values (control characters used to break requests) ** Easier installation: All database-dependent packages can be installed by Python itself ** DB connection setting "encoding" removed (will be detected automatically) ** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly === Version 3.7.4 [2021-05-17] === * Last version for Python 2! * Fixes ** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension). ** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads). ** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly. ** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map). === Version 3.7.3 [2019-12-02]=== * Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards). * ABCD 2.1 template in manual query form moved right next to ABCD 2.06 * Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed. * Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters. === Version 3.7.2 [2019-07-10]=== * Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed. * Bugfixes in Local Query Tool ** Error page on "Grouping page" now displayed correctly. ** for SSL handshake error on certain installations if HTTPS is used. ** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored). * PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations. * QueryTool: Preview added in detail view for multimedia types video and sound * Template for connecting to an [https://www.irisbg.com IrisBG database] added. === Version 3.7.1 [2018-10-02] === * XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source. * New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory. * GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy. * Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation). * Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory). === Version 3.7 [2018-06-12]=== * Dockerfile and Makefile added for dockerisation * ODBC modules now use pyodbc or ceODBC, whichever is installed. * DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval. * Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status. * Java version now shows type of JRE installed (for example OpenJDK) * If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files). * Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence. * Default value for graphviz path in config.ini set to Debian default for easier dockerisation. * Apache snippet in setup script adapted to Apache 2.4 * Fully functional demo data source included (uses included Sqlite database). * Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form. * Fixes ** in dbmod_oracle for metadata retrieval ** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs. ** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed. ** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded. ** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+). ** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty). === Version 3.6.4 [2017-07-04] === * QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation. * ''System administration'' page now allows to view log files in browser * Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed * Fixes ** Bugfix in wrapper core for multi-column primary keys ** Bugfix for "remove cached objects" in system administration ** Bugfix in QueryTool for search fields with no label defined (XPath used instead now) ** Bugfix in metadata retrieval for dbmod_pymssql_2012+ === Version 3.6.3 [2016-08-08] === * DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided. * Fix/change for dbmod_oracle.py: Paging is now faster * New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file === Version 3.6.2 [2016-07-21] === * Debug output now contains INFO items on time required for SQL execution and XML creation. * Changes in DarwinCore transformation: ** Transformation streamlined ** ''IdentificationQualifier'' is now correctly filled from ABCD field ** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped) ** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core) ** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification ** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name * Fixes ** Bugfix in metadata retrieval for dbmod_pymssql ** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL === Version 3.6.1 [2016-03-14] === * New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive). * DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified. * Fixes ** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given). ** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini). === Version 3.6.0 [2015-10-27] === * Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core). * Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements) * XML archiving improved ** uses less memory now (and will be probably faster with larger datasets) ** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period. * Support for ABCD 2.1 added to QueryTool and archiving. * ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN). * DB mod for Excel not experimental any more. * Fixes ** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''. ** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension ** Namespace only added for all attributes if attributeForm=qualified in schema definition. ** base_dbmod: verifyTable now retrieves the table list from the DB only once. ** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly. ** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014. ** Warning ''Destination hostname could not be determined'' removed. ** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software. ** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead. ** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias. === Version 3.5.3 [2015-01-23] === * QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available. * New experimental dbmod for Excel (ODBC). * Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool. * Fixes ** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field). ** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue. ** CMF upgrades now also keep the static table definition. === Version 3.5.2 [2014-07-18] === * Fixes ** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations). ** Admin password removed from configuration dump in system administration. ** For update hint (for the case that an installation was updated before the hint advised this). ** DwC archiving *** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive). *** EML document now also valid if some metadata elements in ABCD are empty. * Namespace prefix for ABCD 2.1 added. * Some content added to dummy files to force Winzip into extracting empty folders from the download archive. * Warning added to table setup if configured PKs/FKs do not exist in database any more. === Version 3.5.1 [2014-04-10] === * DwC Archives ** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added. ** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber''). ** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code. * DB connectivity ** Paging mechanism optimised for Oracle (rownum replaced with row_number). ** Unicode handling changed for MySQL. ** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging. ** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented. ** When Postgres is used, another port than the default 5432 can be used. * Update Mechanism ** Provider Software checks for updates once a week and shows update hint if a new version is available. ** Installation script now supports importing the configuration of an old installation, including mapped data sources. * QueryForm now also shows Dataset Inventory. * Testlibs page now shows detailed error message in hover text for packages that are not installed. * Data source lists are now sorted alphabetically. * Fixes ** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used. ** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed. ** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row. === Version 3.5 [2013-12-05] === * XML archiving now shows progress in percent. * Mapping editor concept tree ** Overhauled (collapsable subtrees thanks to Falko Glöckler) ** Fix for attributes (used to be misplaced when ''show all concepts'' was checked) ** Now has a floating ''Save'' button * New dbmods for SQL Server 2012 added, supporting new OFFSET clause. * Fixes ** Term ''recordedBy'' is now filled correctly in DarwinCore archives. ** Mapping editor now works even if there is no connection to the database. ** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title. ** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title. * Cosmetics ** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix. === Version 3.4.2 [2013-07-15] === * XML Archiving can now be limited to certain datasets within one data source. * Optional attribute ''expires'' for archives added to dataset inventory. * Fixes: ** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer). ** Archiving page now also works with Safari. ** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum. ** Namespace prefixes added for DNA and EFG extensions. ** More verbose error messages for dbmods odbc_access and odbc_tsql. === Version 3.4.1 [2013-04-04] === * Fixes: ** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys. ** in dbmod_mysql (MySQL) for unicode databases. ** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record. ** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key). ** for non-ASCII characters in element attributes (used to break the request). === Version 3.4 [2013-02-20] === * XML archiving re-implemented: ** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance. ** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives. ** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive. * DwC archive transformation simplified. * Archiving page overhauled. * Archiving information shown in datasource configuration page overhauled. * New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5. * Corrections in layout of DB structure page in datasource configuration tool. * Default record limit for web service set to 1000. * Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''. * Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes. * Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document). * Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access. * Fix for dbmod_orcale: Synonyms for tables now included in table lists. * Fix for dbmod_odbc_access: Python error when SQL statement failed removed. * Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved. === Version 3.3.2 [2012-11-26]=== * Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present. * Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed. === Version 3.3.1 [2012-11-09] === * Archiving: Archive name is now set automatically by BioCASe; various smaller changes. * Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added. * Fixes ** dbmod_odbc_tsql now lists tables of all schemas (not only dbo) ** dbmod_oracle/oracle now use correct format for date/datetime conversion ** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases) ** Bugfix for conversion of XML data type "date" === Version 3.3 [2012-10-04] === * Archiving can now be triggered from outside the UI through a deep link (allows scheduling) * Datasource Overview now lists DarwinCore Archives; several smaller changes in UI. * Bugfixes ** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's. ** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes. ** Dummy files added to empty folders to prevent WinZip from ignoring them. ** Local QueryTool: Umlauts in scientific names are now replaced with placeholder * === Version 3.2 [2012-07-09] === * DarwinCore Archives fully integrated into config tool interface * Changes for XML archiving: ** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file) ** Log files of failed or cancelled archiving runs can be viewed later ** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used) ** Format of output changed * Fixes for page ''DB Structure'': ** DB graph generation works now also when IIS is used ** No exception thrown in case there is no connection to the database * Other fixes: ** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change) ** install/readme files updated === Version 3.1 [2012-04-23] === * Fixes for XML archives: ** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html) ** Webinterface overhauled; reacts faster now because server interaction has been reduced ** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress ** XML declaration added to documents ** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added ** Bugfix for the case that no results are returned for the chosen mapping. ** Archiving binary moved from archive working folder to archive lib folder * Metadata retrieval added for pymssql (SQL Server connectivity package) * Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed) * "Wrong" error messages removed for unicode datasources and in mapping editor * Setup script now fixes write permissions for archive folders (working/download directory) * Hint on the usage of literals and concatenation added to mapping editor * Bugfix in library test page (for the case that Apache has different access privileges than Python) * Bugfix in Mapping Editor for literals containing unicode characters * Bugfixes for deleting a datasource: ** Archive folders will now be correctly removed (both working and download directory) ** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin * Bugfix for start page: Leftover debug code removed that always logged an error ("in index") * Bugfix for IIS: Error message "No handlers could be found for logger lib" removed ===Version 3.0.1 [2012-04-05]=== replaced by 3.1 ===Version 3.0 [2011-12-14]=== * New feature: XML Archives (integrated into UI), DwC Archives (separate application) * List of table/column names now also available for Oracle databases * "Report a bug" links added * Fixes in wrapper core: ** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used) ** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order) * Optimization in wrapper core: ** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query. ** Db_mod Oracle now uses paging optimization. ===Version 3.0RC2 [2011-11-24]=== ===Version 3.0RC1 [2011-11-01]=== == Version 2 History == ===Version 2.6.1 [2011-08-23]=== * Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm. * ''Help'' links now point to the respective sections of the new PyWrapper Wiki. * Link to FAQ added to the encoding error page (which can be used to alter system enocding). * Non-existing upgrade function removed from setup script. * Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary. * Bugfix in XML Handler for unary operators (isNull, isNotNull). ===Version 2.6.0 [2011-06-07]=== New Features: * Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used. * When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably. * Support for DNA extension of ABCD (data model for the DNA Bank network) added * Mapping Editor shows XML Path for current element * Deleted datasources and schemas will now be moved to a recycle bin and can be restored later. * Several confirmation dialogs and smaller changes in UI. * mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC * "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable. Bugs fixed: * Session management: User gets logged out only when inactive for more than 30 mins. * Bugfix for unicode characters in mapped literals. * Mapping editor: Added elements do not disappear temporarily any more. * Subversion users: BPS now can handle write-protected svn folders when removing a datasource. * Graphviz dot binary now works also when installed in a folder with a name containing spaces * Creating multiple mapping files for same namespace will be prevented now * Display issues for paths containing "\b" and "\t" * Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR. === Version 2.5.4 [2010-09-14] === Withdrawn; changes included in version 2.6 === Version 2.5.3 [2010-05-12] === * Support for HISPID5 added * Bugfix CMF upgrade * Bugfix QueryTool (Stylesheet transformation error) === Version 2.5.2 [2009-10-01] === * Bugfix QueryTool (Unicode conversion error) === Version 2.5.1 [2009-01-25] === * Security Bugfix (database credentials removed from debug output) === Version 2.5.0 [2008-04-29] === * replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore) * removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one. * Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder! * remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far) * Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely * When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2 * Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging * new sesion based login that stays * updated setup.py script * changed the library test page * and last but not least for developers: cleaned up quite some code === Version 2.4.2 [2006-11-16] === * configtool mapping pane. use anchors to go back to last editing position * typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it. * ABCD2.06 CMF template updated to contain the allowed enumeration of ranks! === Version 2.4.1 [2006-06-26] === * Football Worldcup 2006 logos included * minor bugfixes (dead links) * admin password management fixed === Version 2.4.0 [2006-04-06] === * move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema * webapps log into rotating files instead of screen * updated CMF parser service link * configtool catches errors for unknown static aliases * querytool XSLTs for abcd & abcdefg2.06 * remove wrapper_config.xml * removed character encoding bug in CMF pickling. === Version 2.3.2 [2006-03-06] === * ABCDEFG extension 1.0 for Earth sciences included * List available character encodings for python2.4 and 64bit systems * QueryTool capable of querying DiGIR providers for details. * new synthesys/biocase QT skin === Version 2.3.1 [2005-11-30] === * new 4th dimension dbmod! * record filters inside CMFs fixed * works with NoDIT v2 again * GCP Passport 1.03 added === Version 2.3.0 [2005-11-03] === * compatible with Python 2.4 /!\ Using a new metakit compilation for windows. * Graphviz's dot is being used to visualize the database structure. * ABCD 2.06 included instead of ABCD2.05 === Version 2.2.4 [2005-10-13] === * include PonTaurus example db * scans without a filter crashed with some dbmods. bug fixed. * use select distinct in access for scans * ConfigTool includes graphical view of the db structure graph through graphviz dot images * CMF filters with scan filters crashed the wrapper. solved. * typeconverter: fixed bug in transforming zero-length strings to integers. === Version 2.2.3 [2005-09-01] === * "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more * abcd2.05 debugform templates included * fixed scan problem with oracle8 module * fixed connection problem with native firebird module * Querytool allows to view raw xml * Querytool can handle unicode queries now * Concatenation with pure whitespace works now in wrapper output * database module is logged again into the protocol header. was broken before * if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately. * configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section === Version 2.2.2 [2005-07-22] === * Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs. * Update MCPD CMF template to include w3c types. * Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service. === Version 2.2.1 === * Alternative root tables for SPICE wrappers work now correctly. * Hierarchy requests in SPICE parse correct ID parameters now. * corrected SPICE1.3 CMF template * full Berlin Model SPICE CMF example === Version 2.2.0 [2005-07-18] === * System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords. * setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations. * SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated. * ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard. * Bugfixes in configtool, mapping section: ** Internet Explorer compatability fixed (no tablealias names were saved) ** CMF filter display fixed. Colored green now ** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly ** Fixed minor display problems with attributes * fixed a serious new bug of the configtool that does not allow to add new tables to the db structure * Allow case insensitive like searches in oracle if wanted. * The performance debug logging works again === Version 2.1.0 === * CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys! * SPICE search: bug removed with genus search including the wildcard * SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed! * optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE * cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name * oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY * attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings * bug in local conditional mapping removed. * logging fixed (error and request) * only use "text" instead of "string" and "unicode" everywhere now in the code. * ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson]. === Version 2.0.3 === * SPICE search: Modify search string for genera only by adding " *" and normalize whitespace. === Version 2.0.2 === * Limit for Oracle via ROWNUM implemented for ora8,9&10 * Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves. * SQLLite module added === Version 2.0.1 === * ABCD 2.0 CMF included Bugfixes * Configtool ** Lot of little bugs solved. ** Make possible to map attributes ** Be able to delete mappings ** Alert user when several CMFs with the same namespace exist for a datasource ** Allow full xpaths to be entered directly into the add path form ** allow datasource admin to delete all serialized objects * QueryTool ** More inteligent XSLT for processing * PyWrapper ** Solved error with wrapper not presenting attributes ** Empty elements are now excluded in the output. === Version 2.0.0 === * SPICE protocol supported for Species2000 CDM 1.20 * completely rewritten config tool including a concept retrieval interface to search for concepts * completely rewritten an adaptable query tool with skin support == Version 1 history == === Version 1.6b === * needs at least version 1.4b of the pywrapper * added pywrapper skeleton for new species2000 protocol * cleaned the querytool sourcecode (thanks to Doreen Pahlke) * added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version. === Version 1.56 === * Removed authorization bug when entering non ascii characters as the password. * added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg === Version 1.55 === * NoDIT v2 metaprofile templates and preconfigured database corrected * added a GBIF registration method in the configtool === Version 1.54 === * added a query engine prototype for CMF concepts to the configtool * more database modules supported (Oracle8, FoxPro) === Version 1.52 === * added URL encoding service in cgi-test. * added AltText to images. * removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu. * Updating the cgi-test script to fully support Unix systems. === Version 1.5 === * Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD. * template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory. * private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private. === Version 1.41 === * first release including the query-tool 36f31b8e9783ecf2865a5e61bb6f76e717a7fa55 0 6 1173 1154 2022-06-10T09:33:01Z JoergHoletschek 3 /* Installing Apache/IIS and Python */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|650px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 3.7+ to run, it doesn't work with Python 2, which ended its life cycle in 2020 (same is true for Python 3 versions < 3.7). On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.) If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_3.8.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|300px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|650px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|550px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|550px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|550px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|550px]] === Installing Additional Python Packages/Java/Graphviz === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, only the BioCASe libraries should be installed, since they're part of the archive file you’ve downloaded (or the Subversion repository you checked out from): [[image:instLibrariesVanilla.png|550px]] If you plan to use the Local Query Tool (a small portal for querying and browsing your BioCASe webservice), you need to install the lxml library. You can do this easily by running the Python Package Installer (PIP): ''pip install lxml''. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, you'll need the MySQLdb library; the table shows you the command to run for installing it using pip: ''pip install mysqlclient''. Once pip has installed the package, reload the libraries test page. The according list entry should turn from an orange ''not installed'' into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link and install the package using an installer file or a package manager on Linux. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). BioCASe requires Java 1.8 or later for creating DarwinCore archives, so please check the version listed on the Testlibs page if it's already installed on the server. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Path to Graphviz executable</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:/Program Files/Graphviz 2.44.1/bin/''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for your DBMS, lxml (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. If you want to connect BioCASe to a variety of different database management systems, create DarwinCore archives and use the QueryTool, the libraries test pages should look like this: [[image:instLibrariesDone.png|650px]] == Updating an existing BioCASe Installation == === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. eed443e74737730ddfd6e42c3e6842781b41aa18 1183 1173 2024-06-11T13:55:21Z JoergHoletschek 3 /* Installing BioCASe from a Docker image */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # If you're not using Linux, install the ''make'' tool from https://gnuwin32.sourceforge.net/packages/make.htm. # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|650px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 3.7+ to run, it doesn't work with Python 2, which ended its life cycle in 2020 (same is true for Python 3 versions < 3.7). On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.) If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading the Provider Software === You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository. ==== Downloading the Archive File ==== Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder. ==== Downloading from the Subversion Repository ==== If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled. The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_3.8.0. In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose <small>Checkout</small>. Fill in the URL, change the destination folder name into something meaningful and press OK: [[image:instTortoiseMenu.png|300px]][[image:instTortoiseDialog.png|450px]] For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|650px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|550px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|550px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|550px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|550px]] === Installing Additional Python Packages/Java/Graphviz === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, only the BioCASe libraries should be installed, since they're part of the archive file you’ve downloaded (or the Subversion repository you checked out from): [[image:instLibrariesVanilla.png|550px]] If you plan to use the Local Query Tool (a small portal for querying and browsing your BioCASe webservice), you need to install the lxml library. You can do this easily by running the Python Package Installer (PIP): ''pip install lxml''. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, you'll need the MySQLdb library; the table shows you the command to run for installing it using pip: ''pip install mysqlclient''. Once pip has installed the package, reload the libraries test page. The according list entry should turn from an orange ''not installed'' into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link and install the package using an installer file or a package manager on Linux. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). BioCASe requires Java 1.8 or later for creating DarwinCore archives, so please check the version listed on the Testlibs page if it's already installed on the server. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Path to Graphviz executable</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:/Program Files/Graphviz 2.44.1/bin/''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for your DBMS, lxml (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. If you want to connect BioCASe to a variety of different database management systems, create DarwinCore archives and use the QueryTool, the libraries test pages should look like this: [[image:instLibrariesDone.png|650px]] == Updating an existing BioCASe Installation == === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. d6b62f4aa6417d609779a5823af69d52db7d0192 1195 1183 2025-02-11T14:57:26Z JoergHoletschek 3 /* Downloading the Provider Software */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # If you're not using Linux, install the ''make'' tool from https://gnuwin32.sourceforge.net/packages/make.htm. # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|650px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 3.7+ to run, it doesn't work with Python 2, which ended its life cycle in 2020 (same is true for Python 3 versions < 3.7). On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.) If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading and unpacking the Provider Software === You can find the latest release of BioCASe on our [http://www.example.com Git page]. There are two archives available for download: * The [ default package (recommended)] contains the full installation and allows the generation of DarwinCore archives. For this feature, the [Kettle libraries] are included in the download. * The [ minimal package] doesn't contain the Kettle libraries, hence it's not possible to create DarwinCore archives with such an installation. Choose this download only if you have a good reason to do so. Download the file to your server and unpack it to the desired location of your file system e.g. ''/opt'' on Linux). It contains the BioCASe Provider Software in a folder that contains the version number, e.g. ''bps-3.8.6''. We recommend to rename it to something that doesn't include the version number, so you don't need to touch your web server's configuration during future updates. For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|650px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|550px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|550px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|550px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|550px]] === Installing Additional Python Packages/Java/Graphviz === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, only the BioCASe libraries should be installed, since they're part of the archive file you’ve downloaded (or the Subversion repository you checked out from): [[image:instLibrariesVanilla.png|550px]] If you plan to use the Local Query Tool (a small portal for querying and browsing your BioCASe webservice), you need to install the lxml library. You can do this easily by running the Python Package Installer (PIP): ''pip install lxml''. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, you'll need the MySQLdb library; the table shows you the command to run for installing it using pip: ''pip install mysqlclient''. Once pip has installed the package, reload the libraries test page. The according list entry should turn from an orange ''not installed'' into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link and install the package using an installer file or a package manager on Linux. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). BioCASe requires Java 1.8 or later for creating DarwinCore archives, so please check the version listed on the Testlibs page if it's already installed on the server. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Path to Graphviz executable</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:/Program Files/Graphviz 2.44.1/bin/''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for your DBMS, lxml (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. If you want to connect BioCASe to a variety of different database management systems, create DarwinCore archives and use the QueryTool, the libraries test pages should look like this: [[image:instLibrariesDone.png|650px]] == Updating an existing BioCASe Installation == === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. 314d469b88e712a65556e9b347ae2aa174956d17 1196 1195 2025-02-11T15:02:59Z JoergHoletschek 3 /* Downloading and unpacking the Provider Software */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # If you're not using Linux, install the ''make'' tool from https://gnuwin32.sourceforge.net/packages/make.htm. # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|650px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 3.7+ to run, it doesn't work with Python 2, which ended its life cycle in 2020 (same is true for Python 3 versions < 3.7). On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.) If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading and unpacking the Provider Software === You can find the latest release of BioCASe on our [https://git.bgbm.org/biocase/bps/-/releases/permalink/latest Git page]. There are two archives available for download: * The [https://git.bgbm.org/biocase/bps/-/archive/3.8.6/bps-3.8.6.zip default package (recommended)] contains the full installation and allows the generation of DarwinCore archives. For this feature, the [Kettle libraries] are included in the download. * The [https://www.biocase.org/products/provider_software/files/bps-3.8.6-minimal.zip minimal package] doesn't contain the [https://github.com/pentaho/pentaho-kettle Kettle libraries], hence it's not possible to create DarwinCore archives with such an installation. Choose this download only if you have a good reason to do so. Download the file to your server and unpack it to the desired location of your file system e.g. ''/opt'' on Linux). It contains the BioCASe Provider Software in a folder that contains the version number, e.g. ''bps-3.8.6''. We recommend to rename it to something that doesn't include the version number, so you don't need to touch your web server's configuration during future updates. For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|650px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|550px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|550px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|550px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|550px]] === Installing Additional Python Packages/Java/Graphviz === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, only the BioCASe libraries should be installed, since they're part of the archive file you’ve downloaded (or the Subversion repository you checked out from): [[image:instLibrariesVanilla.png|550px]] If you plan to use the Local Query Tool (a small portal for querying and browsing your BioCASe webservice), you need to install the lxml library. You can do this easily by running the Python Package Installer (PIP): ''pip install lxml''. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, you'll need the MySQLdb library; the table shows you the command to run for installing it using pip: ''pip install mysqlclient''. Once pip has installed the package, reload the libraries test page. The according list entry should turn from an orange ''not installed'' into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link and install the package using an installer file or a package manager on Linux. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). BioCASe requires Java 1.8 or later for creating DarwinCore archives, so please check the version listed on the Testlibs page if it's already installed on the server. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Path to Graphviz executable</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:/Program Files/Graphviz 2.44.1/bin/''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for your DBMS, lxml (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. If you want to connect BioCASe to a variety of different database management systems, create DarwinCore archives and use the QueryTool, the libraries test pages should look like this: [[image:instLibrariesDone.png|650px]] == Updating an existing BioCASe Installation == === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. 0362ac1034bab5b361680762272f4deecb6e0640 1197 1196 2025-02-11T15:03:55Z JoergHoletschek 3 /* Downloading and unpacking the Provider Software */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # If you're not using Linux, install the ''make'' tool from https://gnuwin32.sourceforge.net/packages/make.htm. # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|650px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 3.7+ to run, it doesn't work with Python 2, which ended its life cycle in 2020 (same is true for Python 3 versions < 3.7). On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.) If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading and unpacking the Provider Software === You can find the latest release of BioCASe on our [https://git.bgbm.org/biocase/bps/-/releases/permalink/latest Git page]. There are two archives available for download: * The [https://git.bgbm.org/biocase/bps/-/archive/3.8.6/bps-3.8.6.zip default package (recommended)] contains the full installation and allows the generation of DarwinCore archives. For this feature, the [https://github.com/pentaho/pentaho-kettle Kettle libraries] are included in the download. * The [https://www.biocase.org/products/provider_software/files/bps-3.8.6-minimal.zip minimal package] doesn't contain the Kettle libraries, hence it's not possible to create DarwinCore archives with such an installation. Choose this download only if you have a good reason to do so. Download the file to your server and unpack it to the desired location of your file system e.g. ''/opt'' on Linux). It contains the BioCASe Provider Software in a folder that contains the version number, e.g. ''bps-3.8.6''. We recommend to rename it to something that doesn't include the version number, so you don't need to touch your web server's configuration during future updates. For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|650px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|550px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|550px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|550px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|550px]] === Installing Additional Python Packages/Java/Graphviz === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, only the BioCASe libraries should be installed, since they're part of the archive file you’ve downloaded (or the Subversion repository you checked out from): [[image:instLibrariesVanilla.png|550px]] If you plan to use the Local Query Tool (a small portal for querying and browsing your BioCASe webservice), you need to install the lxml library. You can do this easily by running the Python Package Installer (PIP): ''pip install lxml''. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, you'll need the MySQLdb library; the table shows you the command to run for installing it using pip: ''pip install mysqlclient''. Once pip has installed the package, reload the libraries test page. The according list entry should turn from an orange ''not installed'' into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link and install the package using an installer file or a package manager on Linux. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). BioCASe requires Java 1.8 or later for creating DarwinCore archives, so please check the version listed on the Testlibs page if it's already installed on the server. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Path to Graphviz executable</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:/Program Files/Graphviz 2.44.1/bin/''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for your DBMS, lxml (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. If you want to connect BioCASe to a variety of different database management systems, create DarwinCore archives and use the QueryTool, the libraries test pages should look like this: [[image:instLibrariesDone.png|650px]] == Updating an existing BioCASe Installation == === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. eb5bdbabfdbc86af6d9ac44e5c387e4379d4b7f9 1198 1197 2025-02-11T15:05:05Z JoergHoletschek 3 /* Downloading and unpacking the Provider Software */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # If you're not using Linux, install the ''make'' tool from https://gnuwin32.sourceforge.net/packages/make.htm. # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|650px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 3.7+ to run, it doesn't work with Python 2, which ended its life cycle in 2020 (same is true for Python 3 versions < 3.7). On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.) If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading and unpacking the Provider Software === You can find the latest release of BioCASe on our [https://git.bgbm.org/biocase/bps/-/releases/permalink/latest Git page]. There are two archives available for download: * The [https://git.bgbm.org/biocase/bps/-/archive/3.8.6/bps-3.8.6.zip default package (recommended)] contains the full installation and allows the generation of DarwinCore archives. For this feature, the [https://github.com/pentaho/pentaho-kettle Kettle libraries] are included in the download. * The [https://www.biocase.org/products/provider_software/files/bps-3.8.6-minimal.zip minimal package] doesn't contain the Kettle libraries, hence it's not possible to create DarwinCore archives with such an installation. Choose this download only if you have a good reason to do so. Download the file to your server and unpack it to the desired location of your file system. It contains the BioCASe Provider Software in a folder that contains the version number, e.g. ''bps-3.8.6''. We recommend to rename it to something that doesn't include the version number, so you don't need to touch your web server's configuration during future updates. For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|650px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|550px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|550px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|550px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|550px]] === Installing Additional Python Packages/Java/Graphviz === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, only the BioCASe libraries should be installed, since they're part of the archive file you’ve downloaded (or the Subversion repository you checked out from): [[image:instLibrariesVanilla.png|550px]] If you plan to use the Local Query Tool (a small portal for querying and browsing your BioCASe webservice), you need to install the lxml library. You can do this easily by running the Python Package Installer (PIP): ''pip install lxml''. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, you'll need the MySQLdb library; the table shows you the command to run for installing it using pip: ''pip install mysqlclient''. Once pip has installed the package, reload the libraries test page. The according list entry should turn from an orange ''not installed'' into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link and install the package using an installer file or a package manager on Linux. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). BioCASe requires Java 1.8 or later for creating DarwinCore archives, so please check the version listed on the Testlibs page if it's already installed on the server. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Path to Graphviz executable</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:/Program Files/Graphviz 2.44.1/bin/''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for your DBMS, lxml (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. If you want to connect BioCASe to a variety of different database management systems, create DarwinCore archives and use the QueryTool, the libraries test pages should look like this: [[image:instLibrariesDone.png|650px]] == Updating an existing BioCASe Installation == === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. c0fcc670fd092c6172865d95c250f97d004f18e1 1199 1198 2025-02-11T15:06:51Z JoergHoletschek 3 /* Downloading and unpacking the Provider Software */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # If you're not using Linux, install the ''make'' tool from https://gnuwin32.sourceforge.net/packages/make.htm. # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|650px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 3.7+ to run, it doesn't work with Python 2, which ended its life cycle in 2020 (same is true for Python 3 versions < 3.7). On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.) If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading and unpacking the Provider Software === You can find the latest release of BioCASe on our [https://git.bgbm.org/biocase/bps/-/releases/permalink/latest Git page]. There are two archives available for download: * The [https://git.bgbm.org/biocase/bps/-/archive/3.8.6/bps-3.8.6.zip default package (recommended)] contains the full installation and allows the generation of DarwinCore archives. For this feature, the [https://github.com/pentaho/pentaho-kettle Kettle libraries] are included in the download. * The [https://www.biocase.org/products/provider_software/files/bps-3.8.6-minimal.zip minimal package] doesn't contain the Kettle libraries, hence it's not possible to create DarwinCore archives with such an installation. Choose this download only if you have a good reason to do so. Download the file to your server and unpack it to the desired location of your file system. It contains the BioCASe Provider Software in a folder that contains the version number, e.g. ''bps-3.8.0''. We recommend to rename it to something that doesn't include the version number, so you don't need to touch your web server's configuration during future updates. For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|650px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|550px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|550px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|550px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|550px]] === Installing Additional Python Packages/Java/Graphviz === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, only the BioCASe libraries should be installed, since they're part of the archive file you’ve downloaded (or the Subversion repository you checked out from): [[image:instLibrariesVanilla.png|550px]] If you plan to use the Local Query Tool (a small portal for querying and browsing your BioCASe webservice), you need to install the lxml library. You can do this easily by running the Python Package Installer (PIP): ''pip install lxml''. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, you'll need the MySQLdb library; the table shows you the command to run for installing it using pip: ''pip install mysqlclient''. Once pip has installed the package, reload the libraries test page. The according list entry should turn from an orange ''not installed'' into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link and install the package using an installer file or a package manager on Linux. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). BioCASe requires Java 1.8 or later for creating DarwinCore archives, so please check the version listed on the Testlibs page if it's already installed on the server. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Path to Graphviz executable</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:/Program Files/Graphviz 2.44.1/bin/''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for your DBMS, lxml (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. If you want to connect BioCASe to a variety of different database management systems, create DarwinCore archives and use the QueryTool, the libraries test pages should look like this: [[image:instLibrariesDone.png|650px]] == Updating an existing BioCASe Installation == === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. 604b4ef1b0647fd03ce7cf4c6906ab738af8ca10 1205 1199 2025-03-10T08:46:05Z JoergHoletschek 3 /* Installing BioCASe from a Docker image */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # If you're not using Linux, install the ''make'' tool from https://gnuwin32.sourceforge.net/packages/make.htm. # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|650px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://<dockerhost>/''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 3.7+ to run, it doesn't work with Python 2, which ended its life cycle in 2020 (same is true for Python 3 versions < 3.7). On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.) If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading and unpacking the Provider Software === You can find the latest release of BioCASe on our [https://git.bgbm.org/biocase/bps/-/releases/permalink/latest Git page]. There are two archives available for download: * The [https://git.bgbm.org/biocase/bps/-/archive/3.8.6/bps-3.8.6.zip default package (recommended)] contains the full installation and allows the generation of DarwinCore archives. For this feature, the [https://github.com/pentaho/pentaho-kettle Kettle libraries] are included in the download. * The [https://www.biocase.org/products/provider_software/files/bps-3.8.6-minimal.zip minimal package] doesn't contain the Kettle libraries, hence it's not possible to create DarwinCore archives with such an installation. Choose this download only if you have a good reason to do so. Download the file to your server and unpack it to the desired location of your file system. It contains the BioCASe Provider Software in a folder that contains the version number, e.g. ''bps-3.8.0''. We recommend to rename it to something that doesn't include the version number, so you don't need to touch your web server's configuration during future updates. For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|650px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|550px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|550px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|550px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|550px]] === Installing Additional Python Packages/Java/Graphviz === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, only the BioCASe libraries should be installed, since they're part of the archive file you’ve downloaded (or the Subversion repository you checked out from): [[image:instLibrariesVanilla.png|550px]] If you plan to use the Local Query Tool (a small portal for querying and browsing your BioCASe webservice), you need to install the lxml library. You can do this easily by running the Python Package Installer (PIP): ''pip install lxml''. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, you'll need the MySQLdb library; the table shows you the command to run for installing it using pip: ''pip install mysqlclient''. Once pip has installed the package, reload the libraries test page. The according list entry should turn from an orange ''not installed'' into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link and install the package using an installer file or a package manager on Linux. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). BioCASe requires Java 1.8 or later for creating DarwinCore archives, so please check the version listed on the Testlibs page if it's already installed on the server. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Path to Graphviz executable</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:/Program Files/Graphviz 2.44.1/bin/''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for your DBMS, lxml (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. If you want to connect BioCASe to a variety of different database management systems, create DarwinCore archives and use the QueryTool, the libraries test pages should look like this: [[image:instLibrariesDone.png|650px]] == Updating an existing BioCASe Installation == === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. adef9e075422ebed6a2c11bf1b76f7626d4bb13c 1206 1205 2025-03-10T08:48:11Z JoergHoletschek 3 /* Installing BioCASe from a Docker image */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # If you're not using Linux, install the ''make'' tool from https://gnuwin32.sourceforge.net/packages/make.htm. # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|650px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://<dockerhost>'' (or ''http://<dockerhost>:8181' if you used port 8181). Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 3.7+ to run, it doesn't work with Python 2, which ended its life cycle in 2020 (same is true for Python 3 versions < 3.7). On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.) If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading and unpacking the Provider Software === You can find the latest release of BioCASe on our [https://git.bgbm.org/biocase/bps/-/releases/permalink/latest Git page]. There are two archives available for download: * The [https://git.bgbm.org/biocase/bps/-/archive/3.8.6/bps-3.8.6.zip default package (recommended)] contains the full installation and allows the generation of DarwinCore archives. For this feature, the [https://github.com/pentaho/pentaho-kettle Kettle libraries] are included in the download. * The [https://www.biocase.org/products/provider_software/files/bps-3.8.6-minimal.zip minimal package] doesn't contain the Kettle libraries, hence it's not possible to create DarwinCore archives with such an installation. Choose this download only if you have a good reason to do so. Download the file to your server and unpack it to the desired location of your file system. It contains the BioCASe Provider Software in a folder that contains the version number, e.g. ''bps-3.8.0''. We recommend to rename it to something that doesn't include the version number, so you don't need to touch your web server's configuration during future updates. For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|650px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|550px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|550px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|550px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|550px]] === Installing Additional Python Packages/Java/Graphviz === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, only the BioCASe libraries should be installed, since they're part of the archive file you’ve downloaded (or the Subversion repository you checked out from): [[image:instLibrariesVanilla.png|550px]] If you plan to use the Local Query Tool (a small portal for querying and browsing your BioCASe webservice), you need to install the lxml library. You can do this easily by running the Python Package Installer (PIP): ''pip install lxml''. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, you'll need the MySQLdb library; the table shows you the command to run for installing it using pip: ''pip install mysqlclient''. Once pip has installed the package, reload the libraries test page. The according list entry should turn from an orange ''not installed'' into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link and install the package using an installer file or a package manager on Linux. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). BioCASe requires Java 1.8 or later for creating DarwinCore archives, so please check the version listed on the Testlibs page if it's already installed on the server. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Path to Graphviz executable</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:/Program Files/Graphviz 2.44.1/bin/''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for your DBMS, lxml (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. If you want to connect BioCASe to a variety of different database management systems, create DarwinCore archives and use the QueryTool, the libraries test pages should look like this: [[image:instLibrariesDone.png|650px]] == Updating an existing BioCASe Installation == === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. f14f1a143987a0850e313c5876f7c38a12b4b171 1207 1206 2025-03-10T08:48:31Z JoergHoletschek 3 /* Installing BioCASe from a Docker image */ wikitext text/x-wiki This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first. == Requirements and Considerations == Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].): # You have administrative privileges on this machine, i.e. the user rights to install new software packages. # It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall. # A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL). In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.''' == Installing BioCASe from a Docker image == Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first. '''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+. For installing BioCASe from the Docker image, please follow these steps: # If you're not using Linux, install the ''make'' tool from https://gnuwin32.sourceforge.net/packages/make.htm. # Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+. # Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine. # Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it. Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead: [[File:DockerInstall.png|650px]] Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://<dockerhost>'' (or ''http://<dockerhost>:8181'' if you used port 8181). Don't forget to [[Installation#Changing_the_Default_Password | change the default password]]. You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''. '''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide]. == Regular BioCASe Installation == This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]). BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you: * If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows. * If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian). === Installing Apache/IIS and Python === If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations. Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that. The BPS requires Python 3.7+ to run, it doesn't work with Python 2, which ended its life cycle in 2020 (same is true for Python 3 versions < 3.7). On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.) If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle. Once you’ve successfully installed Apache and Python, you can proceed to the next step. === Downloading and unpacking the Provider Software === You can find the latest release of BioCASe on our [https://git.bgbm.org/biocase/bps/-/releases/permalink/latest Git page]. There are two archives available for download: * The [https://git.bgbm.org/biocase/bps/-/archive/3.8.6/bps-3.8.6.zip default package (recommended)] contains the full installation and allows the generation of DarwinCore archives. For this feature, the [https://github.com/pentaho/pentaho-kettle Kettle libraries] are included in the download. * The [https://www.biocase.org/products/provider_software/files/bps-3.8.6-minimal.zip minimal package] doesn't contain the Kettle libraries, hence it's not possible to create DarwinCore archives with such an installation. Choose this download only if you have a good reason to do so. Download the file to your server and unpack it to the desired location of your file system. It contains the BioCASe Provider Software in a folder that contains the version number, e.g. ''bps-3.8.0''. We recommend to rename it to something that doesn't include the version number, so you don't need to touch your web server's configuration during future updates. For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now: [[image:instBiocaseFolder.png|650px]] === Running the Setup Script === The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later. In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable): [[image:instSetupScriptStart.png|550px]] The questions asked by the setup script do have defaults. If you want to keep them, just press <small>Return</small>. <strong>Webserver domain:</strong> This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later. <strong>Base URL:</strong> This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example. After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window! [[image:instSetupScriptDone.png|550px]] === Linking BioCASe and Your Web Server === The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''. ==== Apache ==== If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes! [[image:instHttpdConf.png|550px]] Save the file and restart Apache for the changes to take effect. ==== Internet Information Server ==== If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (<small>Virtual Directory > Properties > Documents > Default</small>). In order to associate Python scripts with Python in IIS 5/6, you need to use the <small>Configure</small> button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing <small>Add Script Map</small>. Note the placeholders ''%s %s'' after the Python executable! [[File:instAddScriptMap.png|330px]] If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example * http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html * http://support.microsoft.com/kb/276494 ==== Testing the Link ==== Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS: [[image:instBiocaseStartpage.png|550px]] === Installing Additional Python Packages/Java/Graphviz === Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use. On the BPS start page, click on <small>Utilities</small>, then <small>library test</small> in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, only the BioCASe libraries should be installed, since they're part of the archive file you’ve downloaded (or the Subversion repository you checked out from): [[image:instLibrariesVanilla.png|550px]] If you plan to use the Local Query Tool (a small portal for querying and browsing your BioCASe webservice), you need to install the lxml library. You can do this easily by running the Python Package Installer (PIP): ''pip install lxml''. For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, you'll need the MySQLdb library; the table shows you the command to run for installing it using pip: ''pip install mysqlclient''. Once pip has installed the package, reload the libraries test page. The according list entry should turn from an orange ''not installed'' into a green version number. Redo this process for all DBMS you want to connect to. The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link and install the package using an installer file or a package manager on Linux. If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). BioCASe requires Java 1.8 or later for creating DarwinCore archives, so please check the version listed on the Testlibs page if it's already installed on the server. === Changing the Default Password === The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password. Go back to the Start page of the BPS (by clicking <small>Start</small> at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose <small>Config Tool</small>, then <small>System Administration</small>. You will then be prompted for a password. Type in the default ''ACDC'', then click <small>Submit</small>. The top of the configuration page you’ll be directed to should look similar to this: [[image:instConfigTool.png|500px]] In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press <small>update config</small> to save the new password to the configuration file. In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled <small>Path to Graphviz executable</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:/Program Files/Graphviz 2.44.1/bin/''. Press <small>update config</small> to save the path. === Checking Your Installation === Before you proceed to the configuration, you should make sure everything is installed correctly: # Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]). # The libraries test page (<small>Utilities > library test</small>) shows you’ve installed the packages for your DBMS, lxml (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz dot (if you decided so). # <small>Status of writeable directories</small> shows a green <small>writeable</small> for all four directories. If not, the path to problematic file will be given – please grant write access to Python. If you want to connect BioCASe to a variety of different database management systems, create DarwinCore archives and use the QueryTool, the libraries test pages should look like this: [[image:instLibrariesDone.png|650px]] == Updating an existing BioCASe Installation == === Updating a BioCASe Docker installation === This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved. === Updating a regular BioCASe installation === '''<u>Important Note</u>''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it! ==== Updating from version 2.5 or later ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version. # Download the new version as described above into a directory with the name you just memorised. # Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. ==== Updating from older versions < 2.5 ==== # Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later. # Download the new version as described above into a directory with the name you just memorised. # [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase''). # [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation. # Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section <small>Server configuration</small> with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press <small>Update config</small>. # Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation. # Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working. 10b09c5d338d0167dfb4461f62211c41c020e614 0 52 1185 1122 2024-08-15T09:26:44Z JoergHoletschek 3 /* Installation */ wikitext text/x-wiki == General == ===What is the BioCASe Provider Software and what can I use it for?=== See the [[BeginnersGuide|Beginner's Guide]]. ===What do I need to use the BioCASe Provider Software?=== See the requirements in the [[BeginnersGuide|Beginner's Guide]]. ===I’m not sure whether the Provider Software is the right choice for me. How do I find out?=== Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for. ===How much is the BioCASe Provider Software?=== It’s Open Source Software, so you can use it for free. === I am not affiliated with any institution. Can I use the Provider Software in spite of that? === Sure, you can also use the Provider Software as a private person. ===Can I use the Provider Software on my operating system?=== I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems. ===I cannot meet all of the requirements for installing the BPS. What options do I have?=== Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host). Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/ipt) you can create a DarwinCore Archive (see http://rs.tdwg.org/dwc/terms/guides/text/index.htm for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD. ===I have some recommendations/bug report. How can I file it?=== You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]]. == Installation == ===After installing BioCASe, I get a "500: Internal Server Error"=== # Make sure you've [[Installation#Running_the_Setup_Script | run the setup script]] after downloading or checking out BioCASe with the right Python version. To check this, open the file ''www/index.cgi'' of your installation in a text editor and check the first line: This is the Python executable used by Python. # Empty the folder ''cache'' of your installation - please remove all folders and files you find here. # If the problem persists, please contact us! ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?=== If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions. ===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.=== You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]). ===I'm using Internet Information Server, and after creating the Virtual Directory for BioCASe, I get a “502.2 - Bad Gateway. The specified CGI application misbehaved by not returning a complete set of HTTP headers” error.=== This happens when you didn't correctly associate .cgi and .py files with Python. Make sure you've linked both files with Python and that the path contains the placeholders ''%s %s'' (see [[Installation#Internet_Information_Server|Linking IIS]]). ===I cannot find the installation package of the database connectivity module for the DBMS and the Python version I am using.=== You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe. ===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?=== Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe. ===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.=== This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later of psycopg2. ===I'm trying to use BioCASe with Postgres; for this, I tried to install psycopg on Windows. But it doesn't work - either the install crashes, or it finishes, but the test libs page still reports psycopg2 as not installed. In the debug log I can find the error message "ImportError: DLL load failed (...)". What's wrong?=== We suspect this is caused by a bug in recent psycopg2 releases (find more [http://psycopg.lighthouseapp.com/projects/62710/tickets/20 here]). On our test installations, we managed to get round this issue by installing pycopg2 with administration rights. Do do this, right-click the installer file and choose "Run as administrator". ===What's the difference between the DBMS packages odbc_tsql and odbc_tsql_2012 and between pymssql/pymssql_2012, respectively?=== The dbmods ending in "2012" use the new OFFSET clause that was introduced in MS SQL Server 2012. This will make paging faster and increase performance for larger datasets. They work only with SQL Server 2012 or later, and if you use SQL Server 2012, you should use them for best performance. Dbmods without a trailing "2012" work for both older versions and SQL Server 2012, but will be slower in paging large datasets. ===I have problems installing GraphViz. Do I really need it?=== If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz. ===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.=== You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled <small>Graphviz dot binary</small>. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press <small>update config</small> to save the path. ===I want to update my BioCASe installation. How do I do this?=== See [[Installation#Updating_an_existing_BioCASe_Installation|Updating an existing BioCASe Installation]]. ===I've updated my installation from a version prior to 2.6. Now, when I enter the DB structure, some of the table and column names are gone, the same happens in the mapping editor.=== Starting with version 2.6, BioCASe retrieves table and column names from the database for some database management systems (namely MySQL, SQL Server, Postgres, Foxpro, Access and Oracle). In older versions, you had to enter table/column names manually, and when you did the setup with such an early version, you probably didn't care for upper/lower cases. Since some DB management systems are case-sensitive, BioCASe can't ignore the differences and won't find these technically incorrect names in the lists retrieved from the database. The respective drop down boxes will therefore be empty. If that happens, just select the correct names from the drop down lists. ===When I try to create an archive, I get a "Permission denied" error. Why is that?=== For creating archives, you need to grant write permissions for the folders ''archive'' and ''www/downloads'' of your BioCASe installation to Python. You can check if permissions are set correctly in the test libs page (Start --> Utilities --> Test Libs). ===I'm trying to install BioCASe on Ubuntu (or another Linux). Even though I've run the setup script and added the snippet to the Apache configuration file, I see source code when I try out my installation instead of the pages described in the documentation=== Apparently CGI is not enabled on your machine. Google how to enable CGI for your distribution and follow the instructions you'll find. ===When trying to run BioCASe after installation, I get an internal server error (500). But the log folder of BioCASe is empty, nothing to find.=== This is probably because BioCASe - or the web server that runs BioCASe - can't write to this folder. Make sure that the web server (user ''www-data'' for Apache) has write permissions for this folder. ===I'm running BioCASe in a Docker container and I can't connect to a local MS SQL Server (on the same machine, but not in a Docker)=== Do not use host.docker.internal:host-gateway (e.g 172.17.0.1) to connect, but the external LAN IP, often like 192.168.0.x or 10.x.x.x. == Debugging == ===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?=== That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other only by their primary keys. ===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.=== That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element. ===I've mapped ABCD2. When testing the web service, I get a negative number for "RecordsDropped" in the response document. What does that mean?=== Same problem as above - it means you have an error in your mapping. Most probably you've mapped a non-repeatable ABCD concept to a table/column in your database that holds several entries per unit (occurrence) record. The BioCASe Provider Software will duplicate the respective unit records to be able to create valid ABCD documents - which is indicated by a negative number of "records dropped". Make sure the cardinality of your database relations match the structure (in respects of element repeatability) of ABCD to avoid this problem. ===I'm using pymssql to connect to a SQL Server database. Char(n) and Varchar(n) strings get truncated to 255 characters - what's wrong?=== This is known limitation of TDS protocol used by pymssql. Just change the data type for columns storing long string to TEXT or create a view that converts Char/Varchar columns to TEXT columns on the fly to work around this issue. ===I'm using pymssql to connect to a SQL Server database, BioCASe is running on a Linux distribution. When I do a Search or Scan request, I don't get any results (0 hits), even though there are definitely records in the database.=== That problem occurs on old versions of pymssql (1.0.2 or earlier) on Linux distributions. Update to version 2.0 or later to solve this. If you still get empty result sets, these posts might be helpful: * http://open-up.cybertaxonomy.africamuseum.be/forum_topic/important_if_you_use_biocase_bps_linux_platform_connect_ms_sql_server * http://code.google.com/p/pymssql/issues/detail?id=76. ===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.=== Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press <small>Save</small> to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages. ===I am using BioCASe on a Windows Server. For some queries with the Local QueryForm, I get an error page that is obviously produced by the web server and not BioCASe. It reads "Error Code: 500 Internal Server Error. The request was rejected by the HTTP filter. Contact the server administrator. (12217)"=== This happens when BioCASe is run behind a Microsoft ISA Server/Microsoft Forefront Threat Management Gateway. To solve this problem, follow the instructions given in this article: http://support.microsoft.com/kb/837865. ===The information that can be found in the log files folder is too sparse – what can I do?=== Turn on debugging in the System Administration. ===How do I find out if my BioCASe web service works right?=== Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more). ===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.=== The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas. ===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.=== That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org''). ===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?=== The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work. ===I'm using BioCASe in a Vagrant Virtual Box on MacOS with Synced Folders. When using the Manual Query Form or the Archiving page, I get an "ImportError: No module named wrapper.errorclasses."=== We don't know exactly what causes this, but apparently it happens when BioCASe is run in a Vagrant Virtual Box on MacOS and Vagrant's synced folder are used. We do have a workaround for this, but it is not easy. Please [[Special:Contact|contact us]] for assistance on this. ===I'm using BioCASe on a Windows server and odbc_tsql to connect to an MS SQL Server. Even though I do get results for a SCAN request, I cannot do a SEARCH (invalid document received).=== Please check your encoding settings in the DB connection configuration. If it is set to UTF_8, try Latin_1 instead. == Trouble Shooting == ===I’ve forgotten the password of my BioCASe installation. What should I do?=== Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''. ===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.=== First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following: * Go to your Python installation (for example ''/usr/lib/python2.5''), * Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines: <pre> import sys sys.setdefaultencoding('utf-8') </pre> * Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect. ===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?=== Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]]. ===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.=== Just [[Special:Contact|contact us]]; we’ll try our best to help you. 4f0226fb779eaf66d2fa0b0d4385b0833ceb93fb